Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography

TL;DR

Mammo-CLIP is a vision-language model trained on mammogram-report pairs, improving data efficiency and robustness in breast cancer detection tasks, with an added interpretability method for report-based spatial explanations.

Contribution

This work introduces Mammo-CLIP, the first VLM trained on mammogram-report data, enhancing CAD performance and interpretability in mammography analysis.

Findings

Strong classification and localization performance on public datasets

Improved robustness and data efficiency comparable to CLIP in CV

Novel Mammo-FActOR for spatial report interpretation

Abstract

The lack of large and diverse training data on Computer-Aided Diagnosis (CAD) in breast cancer detection has been one of the concerns that impedes the adoption of the system. Recently, pre-training with large-scale image text datasets via Vision-Language models (VLM) (\eg CLIP) partially addresses the issue of robustness and data efficiency in computer vision (CV). This paper proposes Mammo-CLIP, the first VLM pre-trained on a substantial amount of screening mammogram-report pairs, addressing the challenges of dataset diversity and size. Our experiments on two public datasets demonstrate strong performance in classifying and localizing various mammographic attributes crucial for breast cancer detection, showcasing data efficiency and robustness similar to CLIP in CV. We also propose Mammo-FActOR, a novel feature attribution method, to provide spatial interpretation of representation with sentence-level granularity within mammography reports. Code is available publicly: \url{https://github.com/batmanlab/Mammo-CLIP}.