Description
Date depot: 26 octobre 2022
Titre: Combining visual and textual information for enhancing pathologic case retrieval systems in radiological practices
Directrice de thèse:
Florence CLOPPET (LIPADE)
Directeur de thèse:
Camille KURTZ (LIPADE)
Domaine scientifique: Sciences et technologies de l'information et de la communication
Thématique CNRS : Images et vision
Resumé: The field of diagnostic imaging in Radiology has experienced tremendous growth both in terms of technological development (with new modalities such as MRI, PET-CT, etc.) and market expansion. This leads to an exponential increase in the production of imaging data, moving the diagnostic imaging task in a big data challenge. However, the production of a large amount of data does not automatically allow the real exploitation of its intrinsic value for healthcare. In modern hospitals, all imaging data acquired during clinical routines are stored in a picture archiving and communication system (PACS). A PACS is a medical imaging technology providing economical storage and convenient access to images from multiple modalities. Digital images linked to patient examinations are often accompanied by a medical report in text format, summarizing the radiologist’s report and the clinical data associated with the patient (age, sex, medical history, report of previous examinations, etc.). The problem with PACS systems is that they were primarily designed for archival purposes and not for image retrieval exploitation. They therefore only allow a search by keywords (name of the patient, date of the examination, type of examination, etc.) and not by pathologies or by content of the image, and they cannot therefore fulfill the function of diagnostic aid when the doctor is confronted with an image of difficult interpretation or of rare pathology. The objective of this PhD project is to combine cutting-edge research in computer vision and AI to implement a method making it possible to query PACS through example images in order to search for images containing similar pathological cases and to benefit radiologists as a potential decision-making aid during hospital routines.
Background and state of the artIn this context, research is being carried out by LIPADE and the Radiology department of the HEGP (Hôpital Européen Georges Pompidou), to propose different approaches to improve these image search systems. The first is the integration of higher-level image descriptors such as annotations / semantic terms in these processes [KUR14]. These terms can be used to describe a significant amount of information about the visual content of images and are directly related to thehigh-level understanding of the content of images. Some semantic terms can also be automatically predicted from the visual content of the image or regions of interest. The second approach consists in integrating relevance feedback mechanisms [KUR2015], approaches which take into account the validation or not of radiologists on the images found by the system to improve performance. The scientific work resulting from this collaboration has shown the potential of content-based image retrieval (CBIR) systems for the field of Radiology, however they remain at the stage of academic research for the moment. To allow this research to be disseminated and to benefit radiologists as a potential decision-making aid, it is now necessary to integrate it directly into the hospital routinetools by combining all the information contained in PACS used on a daily basis by the experts when novel studying pathological cases.
Contributions of the work to the state-of-the-art Historically, searches were carried out by keywords, and then extended to the search by the example where one wishes to find images visually similar to an example given in image query. To do this, the images are described by their visual characteristics (e.g. gray levels, texture) deduced directly from their pixels and / or from regions of interest delimited in the images. A distance measure is used to find similar images in feature space. Nowadays, state-of-the-art approaches related to most of the sub-tasks underlying to CBIR are based on deep learning approaches. They led to promising results both for textual radiological report mining and for visual image search. In practice, these two data sources are extremely complementary (for example taking into account a priori that a patient is a smoker in order to search for similar lung scans) but their combination within the framework of a CBIRprocess remains complex since these data are not structured in the same way. In a previous project (M2 internship funded by diiP in 2021), we have shown that it was possible to learn efficiently discriminant visual representations of medical images from text supervision. In particular, we were able to achieve such results by adapting the CLIP framework (based on a contrastive learning that uses positive pairs of image and text ), initially proposed for natural images, to the medical / clinical domain . In this PhD project, we aim to pursue / consolidate this workand to investigate the extension of this method to pathological case retrieval, in order to take into account multi-parametric or multi-modal image queries in the retrieval. The query will no longer consist of a single image, but of sets of images.
Doctorant.e: Wei Xiaoyang