MultiSense · System Identification of the Dynamics of Multisensory Integration

System Identification of the Dynamics of Multisensory Integration

The central nervous system (CNS) processes sensory information obtained through various sensory structures in the body. These include signals with a variety of spatiotemporal features, such as different speed and propagation patterns. For instance, visual information moves at the speed of light, enabling a direct spatial representation of the external world, while acoustic signals are far slower, and spatial cues are indirectly computed from amplitude and phase information. In this stream of multisensory information, CNS must decide what information is reliable, what should be rejected and how it should integrate these signals to construct a unique representation of the environment. How CNS accomplishes this process is not known. There are two main aspects of the importance of the project idea for society. First, the results of this work enabled us to increase our understanding of the dynamics of the multisensory integration process. Note that the deficits in multisensory integration in humans are common and often detrimental to the quality of life. Progress on this front will undoubtedly facilitate better diagnosis and treatment for various disorders associated with multisensory integration issues, such as autism spectrum disorder, schizophrenia, and mild cognitive impairment. More importantly, understanding the mechanisms of dynamic sensory reweighting in relation to sensory salience is critical, and it will find implementations in robotics and artificial intelligence. Currently, engineers know how to build custom filters for optimal sensor fusion. However, what is not known is how to perform online and real-time sensory weighting under dynamically changing sensory information. In other words, what is the optimal strategy for tuning the weights associated with each sensory system when the quality of the sensory information changes? Motivated by this problem and the expected impact, our project had four main objectives, involving a systematic combination of behavioral experiments to reveal the underlying dynamics of multisensory integration: (1) Design and build an augmented reality setup that allows controlled and independent multisensory stimulation for the fish during their free-swimming behavior, (2) Identification of the tracking response of the fish during free swimming as a baseline for our multisensory conflict experiments, (3) Independent identification of sensory weights assigned to visual and electrosensory systems by the CNS during refuge tracking behavior, and (4) Monitoring how the CNS tunes its sensory weights in response to varying sensory salience. During the implementation of the project, we developed a unique experimental setup that allows independently stimulating different sensory systems of weakly electric fish and zebrafish during their unconstrained, free behaviors. The setup allows online and real-time data collection under different experimental settings to support the use of control-theoretic analysis, such as data-driven system identification to reveal the dynamics of multisensory integration in freely behaving animals. The infrastructure we’ve developed in this project is modular and can be quickly adapted to use for different problems in neuroscience, where control-theoretic approaches would be a better fit for the analysis. We succeed to estimate the weights assigned to each sensory system independently using control-theoretic approaches. Our analysis allowed us to identify the relative contribution of each sensory modality to the behavioral response of the fish through control-theoretic modeling of the behavior. More critically, we repeated this process under different sensory salience conditions to monitor how these weights are changing as the quality of the sensory information changes. This is critical to characterize the adaptive tuning strategy used by the animals to achieve optimal sensory integration under varying environmental conditions.

Data: CORDIS, © European Union