Description
Date depot: 12 octobre 2023
Titre: Leveraging language models and reinforcement learning for generating instructions and interacting with robots
Directrice de thèse:
Laure SOULIER (ISIR (EDITE))
Directeur de thèse:
Nicolas THOME (ISIR (EDITE))
Domaine scientifique: Sciences et technologies de l'information et de la communication
Thématique CNRS : Intelligence artificielle
Resumé: This PhD is founded by the PILLAR European project (2023-2027) which aims at developing a new generation of robots that can build on the experience acquired during the robots’ lifetime to fulfill the wishes of their human designers/users in real-life applications.
Autonomous agents require reasoning and planning strategies for performing tasks. The semantics captured by large language models can enhance the decision process at different levels. Natural language can serve for building and clarifying the planning strategy, and therefore the actions done by a robot. Several works have addressed instruction identification as abstract representation or natural language expression, but the limited data supervision is often a challenge.
In this thesis, we envision working on the generation of natural language instructions and improving current models. Our objective is to enhance the semantics behind objects to identify the most relevant actions/sub-actions and design hybrid models combining reinforcement learning and language models to generate accurate instructions. We envision three main research challenges: (1) Multi-modal representation learning for instructions, (2) Hybridizing RL and LLMs for richer instruction-following agents, and (3) Proactive interaction for solving uncertainty in instruction generation.
Doctorant.e: Aissi Mohamed Salim