MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

TL;DR

MedXpertQA is a comprehensive, expert-level medical reasoning benchmark with multimodal questions, designed to rigorously evaluate advanced medical knowledge and reasoning capabilities of AI models.

Contribution

The paper introduces MedXpertQA, a new challenging benchmark with multimodal and specialty-specific questions, addressing limitations of previous datasets and including expert validation.

Findings

Evaluated 18 models on MedXpertQA showing varying reasoning abilities.

MedXpertQA's multimodal and specialty-focused design enhances assessment of medical AI.

Benchmark sets a new standard for expert-level medical AI evaluation.

Abstract

We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning. MedXpertQA includes 4,460 questions spanning 17 specialties and 11 body systems. It includes two subsets, Text for text evaluation and MM for multimodal evaluation. Notably, MM introduces expert-level exam questions with diverse images and rich clinical information, including patient records and examination results, setting it apart from traditional medical multimodal benchmarks with simple QA pairs generated from image captions. MedXpertQA applies rigorous filtering and augmentation to address the insufficient difficulty of existing benchmarks like MedQA, and incorporates specialty board questions to improve clinical relevance and comprehensiveness. We perform data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability. We evaluate 18 leading models on \benchmark. Moreover, medicine is deeply connected to real-world decision-making, providing a rich and representative setting for assessing reasoning abilities beyond mathematics and code. To this end, we develop a reasoning-oriented subset to facilitate the assessment of o1-like models. Code and data are available at: https://github.com/TsinghuaC3I/MedXpertQA