Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

TL;DR

This paper introduces DermCase, a comprehensive benchmark for evaluating diagnostic reasoning in rare skin diseases using multimodal data, revealing significant gaps in current vision-language models' capabilities.

Contribution

The paper presents DermCase, a new long-context benchmark with clinical reasoning chains for rare skin diseases, and evaluates LVLMs, highlighting their limitations and potential improvements.

Findings

LVLMs show poor diagnosis accuracy on rare skin diseases

Instruction tuning improves model performance significantly

Current models have critical reasoning limitations

Abstract

Large vision-language models (LVLMs) demonstrate strong performance in dermatology; however, evaluating diagnostic reasoning for rare conditions remains largely unexplored. Existing benchmarks focus on common diseases and assess only final accuracy, overlooking the clinical reasoning process, which is critical for complex cases. We address this gap by constructing DermCase, a long-context benchmark derived from peer-reviewed case reports. Our dataset contains 26,030 multi-modal image-text pairs and 6,354 clinically challenging cases, each annotated with comprehensive clinical information and step-by-step reasoning chains. To enable reliable evaluation, we establish DermLIP-based similarity metrics that achieve stronger alignment with dermatologists for assessing differential diagnosis quality. Benchmarking 22 leading LVLMs exposes significant deficiencies across diagnosis accuracy, differential diagnosis, and clinical reasoning. Fine-tuning experiments demonstrate that instruction tuning substantially improves performance while Direct Preference Optimization (DPO) yields minimal gains. Systematic error analysis further reveals critical limitations in current models' reasoning capabilities.