DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?

TL;DR

DrVD-Bench is a comprehensive multimodal benchmark designed to evaluate whether vision-language models genuinely reason like human clinicians in medical image diagnosis, highlighting current models' limitations in complex reasoning tasks.

Contribution

This work introduces DrVD-Bench, the first structured benchmark for clinical visual reasoning in medical imaging, covering diverse tasks, modalities, and diagnostic categories.

Findings

Performance drops with increased reasoning complexity.

Models often rely on superficial correlations rather than true understanding.

Some models show traces of human-like reasoning but lack grounded visual comprehension.

Abstract

Vision-language models (VLMs) exhibit strong zero-shot generalization on natural images and show early promise in interpretable medical image analysis. However, existing benchmarks do not systematically evaluate whether these models truly reason like human clinicians or merely imitate superficial patterns. To address this gap, we propose DrVD-Bench, the first multimodal benchmark for clinical visual reasoning. DrVD-Bench consists of three modules: Visual Evidence Comprehension, Reasoning Trajectory Assessment, and Report Generation Evaluation, comprising a total of 7,789 image-question pairs. Our benchmark covers 20 task types, 17 diagnostic categories, and five imaging modalities-CT, MRI, ultrasound, radiography, and pathology. DrVD-Bench is explicitly structured to reflect the clinical reasoning workflow from modality recognition to lesion identification and diagnosis. We benchmark 19 VLMs, including general-purpose and medical-specific, open-source and proprietary models, and observe that performance drops sharply as reasoning complexity increases. While some models begin to exhibit traces of human-like reasoning, they often still rely on shortcut correlations rather than grounded visual understanding. DrVD-Bench offers a rigorous and structured evaluation framework to guide the development of clinically trustworthy VLMs.