CEMECS · Cerebral mechanisms underlying the processing of Cued speech information in deaf people: Brain imaging and computational approaches

Final Activity Report Summary - CEMECS (Cerebral mechanisms underlying the processing of Cued Speech information in deaf people: Brain Imaging and Computational approaches)

Cued Speech (CS) is a relatively new (visual) communication system among deaf people that has been introduced three decades ago (Cornett, 1967). It uses hand shapes placed in different positions near the face in combination with the natural mouth movements of speech to make the sounds of spoken language look different from each other. In the speech of hearing individuals, auditory and visual speech percepts are combined, whereas in CS, cued signs and lip reading are both visually perceived. The question thus was how both 'modes' of information are integrated to yield a single percept. We were interested in the cognitive processes underlying CS. Without hearing the sounds, many syllables (the basic speech/sound units used in CS) remain ambiguous in lip reading due to their similar articulatory movements. For example, the consonants in /ma/ and /ba/ or the lip rounding of the vowels in /ku/ and /ko/ cannot be differentiated. A manual cue in this system is made up of two components: the shape of the hand (finger configuration) and the position of the hand near the face. Hand shapes are designed to distinguish among consonants and hand positions among vowels. Visually similar phonemes are coded by perceptually distinctive manual cues, so that a manual cue may correspond to a group of visually contrastive phonemes. In this way, hand shapes and hand positions are used to disambiguate syllables in CS. Spanish possesses a transparent phonological code (5 vocalic sounds, 25 consonantal and semi-vocalic sounds), it is thus particularly suitable for the syllabic gestures of CS. In different studies, we investigated whether standard experimental findings within the field of cognitive psychology could also be found in CS. The production data of abstract (e.g., "liberty", "hate") and concrete words (e.g., "car", "bird") was analysed to examine whether abstractness and concreteness show similar effects in CS production as in 'oral' speech production. Concrete words usually show a processing advantage when compared with abstract words. Results showed that the onset of the first consonant was produced significantly faster in concrete words than in abstract words. However, word frequency did not affect cue production. The finding of a concreteness effect in CS indicates that the motor program of hand gestures becomes part of language representation. That is, the manual cues of CS do not (only) supplement phonemic information but also convey the conceptual information of words. Together, this study shows that concrete concepts may be related not only to the perceptual input but also to expression via learned motor output of the hands, as in the case of CS. In the speech of hearing individuals, auditory and visual speech percepts are combined, whereas in CS, cued signs and lip reading are both visually perceived. The question thus is how both 'modes' of information are integrated to yield a single percept. In Spanish "hearing" speech, high-frequency syllables were found to be produced faster than low-frequency syllables (Carreiras & Perea, 2004; Perea & Carreiras, 1995). This was interpreted to indicate that high-frequency syllables are stored entities that provide fast access for the motor programs during speech production. According to the findings in hearing studies, hand gestures of high-frequency syllables are expected to be produced faster than those of low-frequency syllables. Given that CS and hearing syllables are produced in different modalities, this comparison provides insight whether the motor program of hand gestures becomes part of the articulatory unit. Results for high- and low-frequency syllable in CS suggest that there is an inverse syllable-frequency in CS. However, we found some inconsistencies in these analyses and we corrected our analysing procedures. The final outcome is still pending.

Data: CORDIS, © European Union