MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark

TL;DR

This paper introduces MMRareBench, a comprehensive benchmark for evaluating multimodal and multi-image clinical reasoning in rare diseases, revealing significant gaps in current model capabilities.

Contribution

It presents the first joint evaluation benchmark for multimodal and multi-image reasoning in rare diseases, including curated data and a systematic assessment of 23 models.

Findings

Medical models lag behind general-purpose models in multi-image tasks.

Treatment planning performance is universally low across models.

Fine-tuning improves diagnosis but reduces multi-image reasoning ability.

Abstract

Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely untested. In rare-disease scenarios, clinicians often lack prior clinical knowledge, forcing them to rely strictly on case-level evidence for clinical judgments. Existing benchmarks predominantly evaluate common-condition, single-image settings, leaving multimodal and multi-image evidence integration under rare-disease data scarcity systematically unevaluated. We introduce MMRareBench, to our knowledge the first rare-disease benchmark jointly evaluating multimodal and multi-image clinical capability across four workflow-aligned tracks: diagnosis, treatment planning, cross-image evidence alignment, and examination suggestion. The benchmark comprises 1,756 question-answer pairs with 7,958 associated medical images curated from PMC case reports, with Orphanet-anchored ontology alignment, track-specific leakage control, evidence-grounded annotations, and a two-level evaluation protocol. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. These patterns are consistent with a capacity dilution effect: medical fine-tuning can narrow the diagnostic gap but may erode the compositional multi-image capability that rare-disease evidence integration demands.