Date depot: 9 avril 2021 Titre: Adversarial Attack and Defenses on Android Malware Detection Systems using Reinforcement Learning. Directeur de thèse: Farid NAIT-ABDESSELAM (LIPADE) Domaine scientifique: Sciences et technologies de l'information et de la communication Thématique CNRS : Non defini Resumé: The current state-of-the-art on Android malware detection systems are based on machine learning and deep learning models. Despite having superior performance, these models are susceptible to adversarial attacks. Previous studies are pertinent to adversarial attacks against static analysis-based malware classifiers which only classify the content of the unknown file without execution. However, since the majority of malware is either packed or encrypted, malware classification based on static analysis often fails to detect these types of files. To overcome this limitation, anti-malware companies typically perform dynamic analysis by emulating each file in the anti-malware engine or performing in-depth scanning in a virtual machine. These strategies allow the analysis of the malware after unpacking or decryption. In this research, we will study different strategies of crafting adversarial samples for dynamic analysis of malware detection. In particular Reinforcement learning (RL) can generate intentional perturbations which can be visualized as a min-max game where a reinforcement learning agent determines a sequence of actions to maximize the return based on the reward function. Adversarial attacks are highly dependent on the attacker’s knowledge about the malware detection system. This knowledge can consist of information about the training dataset, features information, model architecture, and classification algorithm used to construct the models. We will also provide defensive methodologies to combat adversarial attacks for dynamic analysis.