MultiMedVision: Multi-Modal Medical Vision Framework

TL;DR

MultiMedVision introduces a unified multi-modal medical imaging framework that processes 2D and 3D data simultaneously using a Sparse Vision Transformer, achieving competitive results with less data.

Contribution

It presents a novel shared encoder architecture for joint 2D/3D medical image representation learning without modality-specific adapters.

Findings

Achieves 0.82 AUROC on MIMIC-CXR and 0.84 on CheXpert for 2D tasks.

Attains 0.85 AUROC on CT-RATE for 3D tasks.

Demonstrates shared and modality-specific features in learned representations.

Abstract

Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance.