A Large Model for Non-invasive and Personalized Management of Breast Cancer from Multiparametric MRI

TL;DR

This paper introduces MOME, a large multimodal model that enhances breast cancer detection and management using multiparametric MRI, matching expert radiologists and supporting personalized treatment decisions.

Contribution

The study presents a novel large mixture-of-modality-experts model that integrates multiparametric MRI data for improved breast cancer diagnosis and treatment prediction, with robustness to missing data.

Findings

MOME matches senior radiologists in accuracy.

Reduces unnecessary biopsies in BI-RADS 4 cases.

Predicts pathological complete response to therapy.

Abstract

Breast Magnetic Resonance Imaging (MRI) demonstrates the highest sensitivity for breast cancer detection among imaging modalities and is standard practice for high-risk women. Interpreting the multi-sequence MRI is time-consuming and prone to subjective variation. We develop a large mixture-of-modality-experts model (MOME) that integrates multiparametric MRI information within a unified structure, leveraging breast MRI scans from 5,205 female patients in China for model development and validation. MOME matches four senior radiologists' performance in identifying breast cancer and outperforms a junior radiologist. The model is able to reduce unnecessary biopsies in Breast Imaging-Reporting and Data System (BI-RADS) 4 patients, classify triple-negative breast cancer, and predict pathological complete response to neoadjuvant chemotherapy. MOME further supports inference with missing modalities and provides decision explanations by highlighting lesions and measuring modality contributions. To summarize, MOME exemplifies an accurate and robust multimodal model for noninvasive, personalized management of breast cancer patients via multiparametric MRI. Code is available at https://github.com/LLYXC/MOME/tree/main.