Description
Date depot: 13 décembre 2021
Titre: Life of a Language Model
Directeur de thèse:
Benoit SAGOT (Inria-Paris (ED-130))
Encadrant :
Eric DE LA CLERGERIE (Inria-Paris (ED-130))
Domaine scientifique: Sciences et technologies de l'information et de la communication
Thématique CNRS : Traitement automatique des langues et de la parole
Resumé: In recent years, neural networks have established themselves as the dominant solution for Natural Language Processing (NLP). However, it is still unclear why they are so successful. Indeed, while methods relying on feature engineering are most of the time very interpretable, neural networks are often compared to black boxes. Their inner weights, optimised during training, offer no insight on why a label is chosen or a word is generated. Even the probabilities associated to each label by the network correlate poorly with true model confidence in a label: models are rarely calibrated. The claim that these networks encode some kind of linguistic structure, although very plausible, can not be verified by simply looking at the model weights.
The idea of probing models stemmed from this observation. Probing aims at uncovering structure or information hidden in neural networks parameters. A large variety of techniques exist for probing. The most common consists in training a small classifier on top of the frozen weights of the model on specific tasks. Probing has shown that models acquire several linguistic skills during training without explicit supervision, and that those skills are effectively useful for prediction at inference time.
The majority of works on evaluating language models (LMs) however consider fixed versions of models, frozen in time. Generally, they are considered at the end of pre-training (in most cases when compute is exhausted), or after fine-tuning. The model then go in its production phase, where its parameters are frozen for inference. We argue that this is a reduced view, and that considering the model at a single timestamp removes a degree of freedom to work with these architectures. For instance early exit, which consists in stopping training prematurely, might be more effective if one could control what quantities have been acquired by the model during training. Similarly, evaluating the model at a time $t$ gives no guarantees that the model will remain up-to-date with the current world after deployment.
In this thesis, we propose to explore what we call the life of a language model. By considering the childhood (pre-training), teenage years (fine-tuning) and adult life (production) of a language model, we hope to provide a larger panorama on how these models evolve and function.
Doctorant.e: Castagné Roman