ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding

TL;DR

ReXInTheWild is a comprehensive benchmark dataset designed to evaluate vision-language models' ability to interpret medical photographs in clinical contexts, highlighting performance gaps and error types.

Contribution

It introduces a new dataset with 955 clinician-verified questions across seven clinical topics, enabling evaluation of multimodal models on medical image understanding and reasoning.

Findings

Leading models achieve up to 78% accuracy.

Performance varies significantly across models.

Error analysis reveals diverse failure modes.

Abstract

Everyday photographs taken with ordinary cameras are already widely used in telemedicine and other online health conversations, yet no comprehensive benchmark evaluates whether vision-language models can interpret their medical content. Analyzing these images requires both fine-grained natural image understanding and domain-specific medical reasoning, a combination that challenges both general-purpose and specialized models. We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics across 484 photographs sourced from the biomedical literature. When evaluated on ReXInTheWild, leading multimodal large language models show substantial performance variation: Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%. A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures and requiring different mitigation strategies. ReXInTheWild provides a challenging, clinically grounded benchmark at the intersection of natural image understanding and medical reasoning. The dataset is available on HuggingFace.