GESTURE · The role of gesture speech synchronisation in interaction and cognition
FP6 — Marie Curie Actions (Human Resources and Mobility)
- Duration
- 2005-09-01 → 2008-08-31
- EU contribution
- €249,805
- Participants
- 1
- Scheme
- OIF
Lines connect the coordinator with its partners. CORDIS does not always give exact coordinates for projects before 2014. These points are placed at city or country level.
Results in brief
Final Activity Report Summary - GESTURE (The role of gesture speech synchronisation in interaction and cognition)
This project investigated how speakers coordinate their words and their gestures in time. In conversation, speakers use hand gestures to provide prosodic, semantic, pragmatic and interactional information. Speakers coordinate their gestures temporally with their speech in specific ways. We investigated how different gestures are aligned with speech. Based on a corpus of video and motion tracking data we determined four different types of temporal relationships providing a systematic account of the temporal relationship of speech and gesture. In addition, we generated animations based on the motion tracking corpus and used them as stimuli investigating if these animations can be used in psychological experiment on audio-visual speech perception. The study showed that the animations are interpreted in similar ways as the video recordings. This provides the basis to be able to exploit motion tracking data for experimentation and the investigation of audio-visual speech perception.
Data: CORDIS, © European Union
Project objective
The main objective of the proposed project is to provide a comprehensive account of gesture-speechsynchronisation in conversational interaction in order to test and extend the scope of existing theories of gesture-speech production. The current theories base their claims on the observation that gesture and speech aretemporally and semantically co-ordinated. This co-ordination is seen as a manifestation of linked cognitiveprocesses underlying speech and gesture production. However, studies conducted within these frameworkssuffer from methodological insufficiencies that call the generalisability of the temporal alignment observationsinto question. Furthermore, despite existing evidence that interactional factors influence gestural behaviour, anddespite the fact that current theories assume that gestures reflect communicative intent, the possible influence ofinteractional factors is not addressed. Given that face-to-face exchanges are the dominant mode ofcommunication, any cognitive theory of gesture-speech production that neglects this dimension is fundamentallyincomplete. The proposed project will expand existing theories in two ways. First, it will provide detailed temporaldata for speech and gesture alignment, covering a range of gesture types, thereby extending thephenomenological scope. Second, the analyses will be based on data elicited from well-controlled interactionaltasks, thereby introducing the interactional dimension of communication. Finally, it will apply a unified andsystematic quantitative analysis to the compiled corpus, introducing a new level of rigor to the field. In sum, theproject will contribute to a unified cognitive processing account of multimodal mechanisms employed in everydayconversation, thereby broadening the scope of the field of psycholinguistics. The proposed project uses motioncapture technology to forge a new paradigm that will provide researchers with a powerful microscope.
Original text from CORDIS.
Participants
- MAX-PLANCK-GESELLSCHAFT ZUR FOERDERUNG DER WISSENSCHAFT VERTRETEN DURCH DAS MAX-PLANCK-INSTITUT FUER PSYCHOLINGUISTIK · NIJMEGENCoordinatorCity levelNetherlands
Links
Data: CORDIS, © European Union