NN-OVEROPT · Neural Network : An Overparametrization Perspective
Horizon 2020 — Marie Skłodowska-Curie Actions
- Duration
- 2021-11-01 → 2024-10-31
- EU contribution
- €257,620
- Participants
- 2
- Scheme
- MSCA-IF
Lines connect the coordinator with its partners.
Results in brief
Neural Network : An Overparametrization Perspective
The problem being addressed: We study the generalization behaviour of training with SGD on convex as well as non-convex models. Importance for society: Our work does not have any direct consequence on society, however, our works make progress towards providing a theoretical foundation for modern machine learning systems. A better theoretical understanding of modern machine-learning systems would eventually lead to better algorithm design across various applications of machine-learning systems. That will result in more interpretable systems which can be further modified to develop bais free algorithms. Overall Objective: In the works, our main goal is to understand optimization and generalization while training a machine learning model with gradient descent, stochastic gradient descent, and noisy gradient descent.
Data: CORDIS, © European Union
Project objective
In recent times, overparametrized models where the number of model parameters far exceeds the number of training samples available are the methods of choice for learning problems and neural networks are amongst the most popular overparametrized methods used heavily in practice. It has been discovered recently that overparametrization surprisingly improves the optimization landscape of a complex non-convex problem, i.e., the training of neural networks, and also has positive effects on the generalization performance. Despite improved empirical performance of overparametrized models like neural networks, the theoretical understanding of these models is quite limited which hinders the progress of the field in the right direction. Any progress in the understanding of the optimization as well as generalization aspects for theses complex models especially neural networks will lead to big technical advancement in the field of machine learning and artificial intelligence. During the Marie Sklodowska-Curie Actions Individual Fellowship-Global Fellowship (MSCA-IF-GF), I plan to study the optimization problem arising while training overparametrized neural networks and generalization in overparametrized neural networks. The end goal for this project is to provide better theoretical understanding of the optimization landscape while training overparametrized models as a result of which to provide better optimization algorithms for training as well as to study the universal approximation guarantees of overparametrized models. We also aim to study the implicit bias induced by optimization algorithms while training overparametrized complex models. To achieve the objective discussed above, I will be using tools from traditional optimization theory, statistical learning theory, gradient flows, as well as from statistical physics.
Original text from CORDIS.
Participants
- INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET AUTOMATIQUE · Le Chesnay CedexCoordinatorFrance
- THE BOARD OF TRUSTEES OF THE UNIVERSITY OF ILLINOIS · UrbanaUnited States
Links
Data: CORDIS, © European Union