IXMI · Improving the eXperience of musicians and spectators with digital Musical Instruments using 3D interfaces and mixed-reality technologies

Improving the eXperience of musicians and spectators with digital Musical Instruments using 3D interfaces and mixed-reality technologies

The increasing diversity and complexity of Digital Musical Instruments opens new opportunities for electronic musicians. However, it also makes electronic performances less and less accessible to the audience. They can't rely on physical properties of the instruments, and musicians' impact on the sound is almost impossible to perceive due to subtle/hidden gestures and complex gestures/sound mappings. Moreover, the familiarity that spectators have with acoustic instruments, which they have either played or seen played, does not exist with digital instruments. The goal of the IXMI project is to use mixed-reality displays and 3D interfaces to augment Digital Musical Instruments in order to reveal their inner mechanisms and the engagement of musicians during performances to the audience. This project was conducted at the University of Bristol by Dr. Florent Berthaut in the Bristol Interaction and Graphics team, between September 2013 and August 2015. The main outcomes of the project are : 1) A novel approach for improving the experience of spectators called Rouages. It is based on the analysis of Digital Instruments according to 3 components and consists in: a) amplifying musicians gestures with virtual extensions of the physical interface b) displaying representations of the various sound processes that compose the instrument (synthesisers, rhythmic and melodic patterns, samples …) c) revealing the links or mappings between the interface and the sound processes, in order to show the spectators which part of the music is directly controlled by the musician's gestures and how, and which part is automated. 2) Novel mixed-reality display technologies that provide consistent visual augmentations for all audience members. a) A study of semi-transparent and shape-changing mirrors for mixed-reality applications b) The Reflets mixed-reality display for artistic performances which allows spectators to explore and perceived consistently visual augmentations of performances using their reflections in a large semi-transparent mirror. Reflets also opens novel musical expression opportunities, as demonstrated during workshops with artists. 3) The evaluation of the perception of performances with Digital Musical Instrument using results from Cognitive Sciences. The IXMI project demonstrates that the perception of the amount of control the musician has over the music can be improved by the addition of visual augmentations. 4) BOEUF: a model of Digital Orchestras that extends the Rouages approach to the visualisation of interactions between musicians These results were presented in publications in Human-Computer Interaction (UIST, ITS) and Computer Music Conferences (NIME, CMMR), during workshops with artists of the local Bristol Scene and public performances in Bristol, UK and Bordeaux, France. The project website is available at http://ixmi.hitmuri.net/ and the software component of the Reflets display can be downloaded at http://forge.lifl.fr/Revil. The IXMI project is now being pursued by Dr Berthaut at the University of Lille with the following objectives: 1) The design of an automatic familiarity system, that generates visual augmentations automatically by learning components of Digital Musical Instruments from practice sessions 2) The creation of a standard audience experience evaluation protocol using physiological sensors, which will allow for finer analysis of the audience perception of performances.

Data: CORDIS, © European Union