STRUDEL · Information Theory beyond Communications: Distributed Representations and Deep Learning
Horizon 2020 — Marie Skłodowska-Curie Actions
- Duration
- 2018-09-01 → 2020-08-31
- EU contribution
- €171,349
- Participants
- 2
- Scheme
- MSCA-IF
Lines connect the coordinator with its partners.
Results in brief
Information Theory beyond Communications: Distributed Representations and Deep Learning
Artificial Intelligence (AI) and deep learning components are present in many of today’s autonomous and intelligent systems and can inevitably affect the safety, assurance, privacy, and performance of these systems interacting with uncertain and dynamic environments. For instance, autonomous cars use deep neural networks to classify and detect obstacles or pedestrians on a road; AI techniques are used in healthcare for diagnosis and in developing algorithms for medical devices; and domestic robots and assistive devices leverage AI algorithms to safely interact with humans. To provide any correctness guarantees for such systems, we first need to understand and formalize the desired, unexpected, or malicious behaviors that could be produced by these systems. These properties may specify the functionality of the inner AI components (e.g., intermediate representations of a multilayer neural network) by defining their input-output behavior. Alternatively, the properties may be at the level of the overall system that encompasses multiple AI components interacting with one another and with other decision-making components. Although some explanations appear to be solidly grounded, there is little mathematical understanding of representation learning. This project capitalizes on powerful and fertile concepts from information theory and information measures in order to advance the state of the art in deep learning. The overall project goal is to develop novel information-theoretic tools and understanding of deep learning based on information measures. The proposed framework is expected to bridge the gap between theory and practice to facilitate a more thorough understanding and hence improved design of deep learning architectures.
Data: CORDIS, © European Union
Project objective
Deep learning is an enormously successful recent paradigm with record-breaking performance in numerous applications. Individual autoencoders (AEs) of a multilayer neural network are trained to convert high-dimensional inputs into low-dimensional codes that allow the reconstruction of the input. Although some explanations appear to be solidly grounded, there is no mathematical understanding of the AE learning process. This project is a collaborative endeavor of researchers with strong complementary backgrounds. Its main innovation is the idea to capitalize on powerful and fertile concepts from information theory (expertise of researcher) in order to advance the state of the art in deep learning (expertise of supervisor at TC). The innovative research work is motivated by our recent insight that there is an intimate relationship between AEs, generative adversarial nets and the information bottleneck method. This method is a model-free approach for extracting information from observed variables that are relevant to hidden representations or labels and will serve as basic building block for an information theory of representation learning. The planned objectives are split into 3 workpackages: 1) information-theoretic criteria and statistical tradeoffs for extracting good representations, 2) structured architectures/algorithms for learning, 3) use of stochastic complexity to assess the descriptive power (model selection) of deep neural networks. Accomplishing the challenging goals of this proposal requires a variety of methodologies with a rich potential for transfer of knowledge between the involved fields of information theory, statistics and machine learning. Our new framework is expected to bridge the gap between theory and practice to facilitate a more thorough understanding and hence improved design of deep learning architectures. The fellow researcher is coordinating the LIA Lab of the CNRS (started in 2017) where he is collaborating with the supervisor at TC
Original text from CORDIS.
Participants
Links
- View on CORDIS
- DOI: 10.3030/797805
- http://webpages.lss.supelec.fr/perso/pablo.piantanida/GroupAux/Reserach_Projects.html
Data: CORDIS, © European Union