brat: Aligned Multi-View Embeddings for Brain MRI Analysis

TL;DR

Brat is a multi-view learning framework that aligns brain MRI embeddings with clinical reports, improving analysis of subtle abnormalities in 3D scans through a large dataset and innovative training methods.

Contribution

The paper introduces brat, a novel multi-view embedding approach for brain MRI analysis, utilizing a large dataset and a new pre-training technique inspired by document retrieval.

Findings

Significant performance improvements across vision-language tasks.

Development of a large, diverse brain MRI dataset with reports.

Public release of brat foundation models.

Abstract

We present brat (brain report alignment transformer), a multi-view representation learning framework for brain magnetic resonance imaging (MRI) trained on MRIs paired with clinical reports. Brain MRIs present unique challenges due to the presence of numerous, highly varied, and often subtle abnormalities that are localized to a few slices within a 3D volume. To address these challenges, we introduce a brain MRI dataset $10\times$ larger than existing ones, containing approximately 80,000 3D scans with corresponding radiology reports, and propose a multi-view pre-training approach inspired by advances in document retrieval. We develop an implicit query-feature matching mechanism and adopt concepts from quality-diversity to obtain multi-view embeddings of MRIs that are aligned with the clinical features given by report sentences. We evaluate our approach across multiple vision-language and vision tasks, demonstrating substantial performance improvements. The brat foundation models are publicly released.