MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents

TL;DR

MedConsultBench is a comprehensive benchmark for medical consultation agents that evaluates the entire clinical workflow, emphasizing process integrity, safety, and detailed inquiry logic, revealing gaps between knowledge and practical clinical skills.

Contribution

It introduces a fine-grained, process-aware evaluation framework with AIUs and metrics, addressing the limitations of prior coarse benchmarks and capturing real-world clinical reasoning.

Findings

High diagnostic accuracy does not ensure effective information gathering.

Models often struggle with medication safety and follow-up inquiries.

The benchmark reveals gaps between AI knowledge and clinical practice skills.

Abstract

Current evaluations of medical consultation agents often prioritize outcome-oriented tasks, frequently overlooking the end-to-end process integrity and clinical safety essential for real-world practice. While recent interactive benchmarks have introduced dynamic scenarios, they often remain fragmented and coarse-grained, failing to capture the structured inquiry logic and diagnostic rigor required in professional consultations. To bridge this gap, we propose MedConsultBench, a comprehensive framework designed to evaluate the complete online consultation cycle by covering the entire clinical workflow from history taking and diagnosis to treatment planning and follow-up Q\&A. Our methodology introduces Atomic Information Units (AIUs) to track clinical information acquisition at a sub-turn level, enabling precise monitoring of how key facts are elicited through 22 fine-grained metrics. By addressing the underspecification and ambiguity inherent in online consultations, the benchmark evaluates uncertainty-aware yet concise inquiry while emphasizing medication regimen compatibility and the ability to handle realistic post-prescription follow-up Q\&A via constraint-respecting plan revisions. Systematic evaluation of 19 large language models reveals that high diagnostic accuracy often masks significant deficiencies in information-gathering efficiency and medication safety. These results underscore a critical gap between theoretical medical knowledge and clinical practice ability, establishing MedConsultBench as a rigorous foundation for aligning medical AI with the nuanced requirements of real-world clinical care.