3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

TL;DR

MedMAP is a novel pretraining framework that improves vision-language alignment and feature fusion in 3D MRI for multi-organ abnormality detection, significantly outperforming existing models.

Contribution

Introduces MedMAP, a modality-aware pretraining approach specifically designed for 3D MRI vision-language tasks, addressing modality-specific challenges.

Findings

MedMAP outperforms existing VLMs on 3D MRI abnormality detection.

Curated MedMoM-MRI3D dataset with 7,392 MRI-volume report pairs.

Effective joint modality distribution capture during pretraining.

Abstract

Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.