Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosis

TL;DR

Med3D-R1 introduces a reinforcement learning framework with a two-stage training process that enhances clinical reasoning and diagnostic accuracy in 3D medical vision-language models, addressing interpretability and overfitting issues.

Contribution

The paper presents Med3D-R1, a novel RL-based training framework with residual alignment and abnormality re-weighting strategies for improved 3D medical diagnosis.

Findings

Achieved state-of-the-art accuracy on CT-RATE and RAD-ChestCT benchmarks.

Enhanced clinical reasoning and interpretability in 3D medical vision-language models.

Outperformed prior methods in abnormality diagnosis tasks.

Abstract

Developing 3D vision-language models with robust clinical reasoning remains a challenge due to the inherent complexity of volumetric medical imaging, the tendency of models to overfit superficial report patterns, and the lack of interpretability-aware reward designs. In this paper, we propose Med3D-R1, a reinforcement learning framework with a two-stage training process: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). During SFT stage, we introduce a residual alignment mechanism to bridge the gap between high-dimensional 3D features and textual embeddings, and an abnormality re-weighting strategy to emphasize clinically informative tokens and reduce structural bias in reports. In RL stage, we redesign the consistency reward to explicitly promote coherent, step-by-step diagnostic reasoning. We evaluate our method on medical multiple-choice visual question answering using two 3D diagnostic benchmarks, CT-RATE and RAD-ChestCT, where our model attains state-of-the-art accuracies of 41.92\% on CT-RATE and 44.99\% on RAD-ChestCT. These results indicate improved abnormality diagnosis and clinical reasoning and outperform prior methods on both benchmarks. Overall, our approach holds promise for enhancing real-world diagnostic workflows by enabling more reliable and transparent 3D medical vision-language systems.