Projet de recherche doctoral numero :8137


Date depot: 7 avril 2021
Titre: Memorization in Deep Learning
Directeur de thèse: Sylvain LAMPRIER (LIP6)
Encadrant : Edouard OYALLON (LIP6)
Domaine scientifique: Sciences pour l'ingénieur
Thématique CNRS : Non defini

Resumé: Deep Neural Networks obtain outstanding performances on many benchmarks, yet the key ingredient of their success remains unknown. This is mainly due to the high dimensional nature of those objects: they have a lot of parameters D and use very large inputs d. By now, without loss in generality, we will focus on Neural Networks Φ learned for a classification task and which have been fed with N samples. The weights of a Neural Network are specified via supervision and those networks tend to generalize well on a new test set: it implies those architectures have memorized important attributes from a dataset. During this PhD, we propose to study those attributes both from a theoretical and numerical point of view: what is their nature, how are they learned, how are they stored? We aims at studying two types of mechanisms which can be addressed independently while being neatly connected: the memorization through the symmetries of a supervised or unsupervised task, and the memorization through the data. Interestingly, any improvement concerning one aspect will benefit on the other aspect.