Strong Reasoning Isn't Enough: Evaluating Evidence Elicitation in Interactive Diagnosis

TL;DR

This paper introduces an interactive evaluation framework and a new benchmark for medical diagnosis, revealing that strong reasoning alone isn't enough for effective evidence collection, and proposes a strategy to improve it.

Contribution

It presents a novel interactive evaluation framework, the EviMed benchmark, and the REFINE strategy to enhance evidence elicitation in medical diagnosis models.

Findings

Strong reasoning doesn't ensure effective evidence gathering.

REFINE improves evidence collection and diagnosis accuracy.

Smaller models can outperform larger ones with proper evidence-guidance.

Abstract

Interactive medical consultation requires an agent to proactively elicit missing clinical evidence under uncertainty. Yet existing evaluations largely remain static or outcome-centric, neglecting the evidence-gathering process. In this work, we propose an interactive evaluation framework that explicitly models the consultation process using a simulated patient and a \rev{simulated reporter} grounded in atomic evidences. Based on this representation, we introduce Information Coverage Rate (ICR) to quantify how completely an agent uncovers necessary evidence during interaction. To support systematic study, we build EviMed, an evidence-based benchmark spanning diverse conditions from common complaints to rare diseases, and evaluate 10 models with varying reasoning abilities. We find that strong diagnostic reasoning does not guarantee effective information collection, and this insufficiency acts as a primary bottleneck limiting performance in interactive settings. To address this, we propose REFINE, a strategy that leverages diagnostic verification to guide the agent in proactively resolving uncertainties. Extensive experiments demonstrate that REFINE consistently outperforms baselines across diverse datasets and facilitates effective model collaboration, enabling smaller agents to achieve superior performance under strong reasoning supervision. Our code can be found at https://github.com/NanshineLoong/EID-Benchmark .