Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis

TL;DR

This paper introduces a self-supervised vision-language alignment method for bone X-ray analysis using French reports, enabling effective downstream tasks with less annotated data and expanding healthcare AI applications.

Contribution

It is the first to incorporate French medical reports into vision-language pretraining for bone X-ray representations, leveraging hospital data for improved medical imaging analysis.

Findings

Competitive performance on osteoarthritis quantification

Effective bone age estimation in pediatric wrists

Successful detection of fractures and anomalies

Abstract

This paper proposes leveraging vision-language pretraining on bone X-rays paired with French reports to address downstream tasks of interest on bone radiography. A practical processing pipeline is introduced to anonymize and process French medical reports. Pretraining then consists in the self-supervised alignment of visual and textual embedding spaces derived from deep model encoders. The resulting image encoder is then used to handle various downstream tasks, including quantification of osteoarthritis, estimation of bone age on pediatric wrists, bone fracture and anomaly detection. Our approach demonstrates competitive performance on downstream tasks, compared to alternatives requiring a significantly larger amount of human expert annotations. Our work stands as the first study to integrate French reports to shape the embedding space devoted to bone X-Rays representations, capitalizing on the large quantity of paired images and reports data available in an hospital. By relying on generic vision-laguage deep models in a language-specific scenario, it contributes to the deployement of vision models for wider healthcare applications.