A Foundation Model for Brain MRI with Dynamic Modality Integration

TL;DR

This paper introduces a versatile foundation model for brain MRI that effectively handles various imaging modality combinations using a unified encoder with modality-aware features, trained on extensive multi-center data.

Contribution

It proposes a novel modality-integrated architecture with learnable embeddings and regularization, enabling flexible MRI analysis without separate models for each modality.

Findings

Feasibility demonstrated on preliminary experiments.

Model adapts to missing or unseen modalities.

Open-source code and pretrained models available.

Abstract

We present a foundation model for brain MRI that can work with different combinations of imaging sequences. The model uses one encoder with learnable modality embeddings, conditional layer normalization, and a masked autoencoding objective that accounts for missing modalities. A variance-covariance regularizer is applied to stabilize feature learning and improve representation diversity. This design removes the need for separate models for each modality and allows the network to adapt when some sequences are missing or unseen. It is trained on about 60,000 multi-center MRIs using self-supervised reconstruction and modality imputation to learn flexible representations. A learnable modality embedding guides feature extraction so the encoder can adjust to different inputs. We describe our planned evaluation on brain tumor and multiple sclerosis segmentation, as well as lesion classification, under various modality settings. Preliminary results show that the method works feasibly, and further experiments are planned to study its performance in more detail. All code and pretrained models are available at https://github.com/BrainFM/brainfm