How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study

TL;DR

This study benchmarks open-source vision-language models on medical tasks, revealing their strengths, limitations, and the gap to clinical deployment, emphasizing the need for improved multimodal alignment and evaluation.

Contribution

It provides a comprehensive evaluation of general-purpose and medical-specific VLMs across diverse benchmarks, highlighting performance gaps and challenges for medical AI applications.

Findings

Large general models match or surpass medical-specific models on some benchmarks.

Reasoning abilities lag behind understanding in model performance.

Performance varies significantly across different medical benchmarks.

Abstract

Vision-Language Models (VLMs) trained on web-scale corpora excel at natural image tasks and are increasingly repurposed for healthcare; however, their competence in medical tasks remains underexplored. We present a comprehensive evaluation of open-source general-purpose and medically specialised VLMs, ranging from 3B to 72B parameters, across eight benchmarks: MedXpert, OmniMedVQA, PMC-VQA, PathVQA, MMMU, SLAKE, and VQA-RAD. To observe model performance across different aspects, we first separate it into understanding and reasoning components. Three salient findings emerge. First, large general-purpose models already match or surpass medical-specific counterparts on several benchmarks, demonstrating strong zero-shot transfer from natural to medical images. Second, reasoning performance is consistently lower than understanding, highlighting a critical barrier to safe decision support. Third, performance varies widely across benchmarks, reflecting differences in task design, annotation quality, and knowledge demands. No model yet reaches the reliability threshold for clinical deployment, underscoring the need for stronger multimodal alignment and more rigorous, fine-grained evaluation protocols.