Menu
MapConnectionsTopicsBulgariaThe Night (K-TRIO)Marie Skłodowska-CurieFor studentsFor teachersMethodologySurprise me
H2020Individual fellowship2016–2018

MULTISIMO · MULTImodal and MULTIparty Social Interactions MOdelling

Horizon 2020 — Marie Skłodowska-Curie Actions

Duration
2016-09-01 → 2018-08-31
EU contribution
€175,866
Participants
1
Scheme
MSCA-IF

Lines connect the coordinator with its partners.

Results in brief

MULTImodal and MULTIparty Social Interactions MOdelling

The goal of the project was to model the multimodal behavior of speakers participating in multiparty social interactions. The target has been the development of new knowledge, i.e. dialog and behavioral models that are indispensable for technologies capable of incorporating human-like communicative abilities into human-computer intelligent interfaces. To this end, this project addressed multimodal and multiparty dialogue management driven by cognitive models of human interactive behavior. The research objective was to understand and model the multiparty interaction configuration and the underlying communicative and social behavior of the participants. This novel research aimed to respond to the following challenges in terms of analysis and modeling: (a) interpretation of conversational behavior and the turn-taking mechanism, (b) detection of personality traits and behavior analysis, and (c) modeling the multimodal strategies and regulatory actions reflected through the participants’ verbal and non-verbal signals and psychological variables. The analysis and interpretation of multimodal signals provided the necessary semantic and communicative context to decode the speakers’ interactional behavior. This research work has been ambitious also in that it goes beyond two-party interaction and will focus on multiparty interaction, a so far under-studied, yet a very common, rich and informative everyday communication phenomenon. Specifically, it has examined and modeled the set of communicative strategies that are important for the participants to acquire, develop and manage, so that they reach a communicative goal. The project also examined the effect of personality traits and emotions on the degree of speakers’ engagement in the interaction, their level of attention and the tendency to create or the ability to manage conversational conflicts. In this context, the achievement of the communicative goal has been assessed through measurable multimodal signals that interlocutors make and has been correlated to the participants’ personality traits and turn-taking activity.

Data: CORDIS, © European Union

Project objective

The project addresses the modelling of multimodal communicative human behaviour during natural interactions in multiparty settings and specifically the conversational and feedback strategies of the interaction participants in terms of attention, engagement and conflict management. The knowledge base for this modelling problem will be a collection of human-human interaction recordings to be acquired through a concise experimental protocol and state-of-the-art equipment. The communicative behavior will be analysed in multimodal signals, including facial expressions, gaze and prosodic cues that provide information about social interaction management and speakers’ attitudes. This information becomes more complex in group interactions, where a management mechanism will be discovered and modelled. These low-level signals will be examined in relation to their timing and sequence to infer and model high-level signals of attention, engagement and conflict, and subsequently implement modules to feed existing HCI interfaces such as dialogue systems or avatars. The implementation of such novel models will allow for the creation of more sophisticated human-like agent-based interfaces of high quality communication experience, especially in scenarios where social and cognitive skills, in addition to the communicative ones, are crucial for the success of the interaction, i.e. in the education, customer care domains and web applications requiring interfaces triggered by human communication. The project builds on the strengths of the applicant in multimodal communication and language technology, and combines these with new training in intelligent systems and human-computer interaction to support a new research direction and to strengthen cross-European collaboration. The host provides an excellent supportive environment conducive to the interdisciplinary research envisaged in the project and the training objectives of the fellowship in view of the applicant’s career development.

Original text from CORDIS.

Participants

  • THE PROVOST, FELLOWS, FOUNDATION SCHOLARS & THE OTHER MEMBERS OF BOARD, OF THE COLLEGE OF THE HOLY & UNDIVIDED TRINITY OF QUEEN ELIZABETH NEAR DUBLIN · DublinCoordinatorIreland

Links

Data: CORDIS, © European Union