Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D

TL;DR

Brain3D introduces a novel 3D vision-language framework for automated brain MRI report generation, significantly outperforming 2D models by utilizing inflated 3D encoders and staged alignment tailored to neuroradiology.

Contribution

The paper presents a new staged training approach for inflating 2D medical encoders into 3D, specifically designed for brain MRI report automation, with improved accuracy and clinical relevance.

Findings

Achieves a Clinical Pathology F1 score of 0.951 on 468 subjects.

Maintains perfect specificity on healthy scans.

Staged alignment is crucial for model performance.

Abstract

Current medical vision-language models (VLMs) process volumetric brain MRI using 2D slice-based approximations, fragmenting the spatial context required for accurate neuroradiological interpretation. We developed \textbf{Brain3D}, a staged vision-language framework for automated radiology report generation from 3D brain tumor MRI. Our approach inflates a pretrained 2D medical encoder into a native 3D architecture and progressively aligns it with a causal language model through three stages: contrastive grounding, supervised projector warmup, and LoRA-based linguistic specialization. Unlike generalist 3D medical VLMs, \textbf{Brain3D} is tailored to neuroradiology, where hemispheric laterality, tumor infiltration patterns, and anatomical localization are critical. Evaluated on 468 subjects (BraTS pathological cases plus healthy controls), our model achieves a Clinical Pathology F1 of 0.951 versus 0.413 for a strong 2D baseline while maintaining perfect specificity on healthy scans. The staged alignment proves essential: contrastive grounding establishes visual-textual correspondence, projector warmup stabilizes conditioning, and LoRA adaptation shifts output from verbose captions to structured clinical reports\footnote{Our code is publicly available for transparency and reproducibility