Structure is Supervision: Multiview Masked Autoencoders for Radiology

TL;DR

This paper introduces MVMAE, a self-supervised learning framework for radiology that exploits multi-view data and reports to improve disease classification, outperforming existing methods.

Contribution

The paper proposes MVMAE and MVMAE-V2T, novel self-supervised models that leverage multi-view radiology data and reports for better medical image representations.

Findings

MVMAE outperforms supervised and vision-language baselines on three datasets.

MVMAE-V2T improves performance especially in low-label settings.

Structural and textual supervision are effective complementary signals.

Abstract

Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision-language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.