Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction

TL;DR

This paper introduces MINT, a multi-turn medical diagnosis benchmark, revealing key behavioral patterns of LLMs and offering strategies to improve their reliability in clinical reasoning tasks.

Contribution

The paper presents MINT, a new benchmark for multi-turn medical diagnosis, and provides insights and recommendations to enhance LLMs' decision-making in clinical settings.

Findings

Models often answer prematurely within the first two turns.

Self-correction occurs more frequently than incorrect-to-correct flips.

Deferring answers and reserving salient evidence improves accuracy.

Abstract

Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn, yet how they behave under multi-turn evidence accumulation closer to real clinical reasoning remains unexplored. We introduce MINT (Medical Incremental N-Turn Benchmark), a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases with clinically labeled evidence shards, controlled turn granularity, and information-preserving decomposition. Through systematic evaluation of 11 LLMs on MINT, we uncover three persistent behavioral patterns that significantly impact diagnostic decisions: (1) intent to answer, models rush to answer before sufficient evidence has been observed, with over 55% of answers committed within the first two turns; (2) self-correction, incorrect-to-correct answer revisions occur at up to 10.6 times the rate of correct-to-incorrect flips, revealing a latent capacity for self-correction that premature commitment forecloses; and (3) strong lures, clinically salient information such as laboratory results trigger premature answering even when models are explicitly instructed to wait. We translate these findings into clinically actionable guidance: deferring the diagnostic question to later turns reduces premature answering and improves accuracy at the first point of commitment by up to 62.6%, while reserving salient clinical evidence for later turns prevents a catastrophic accuracy drop of up to 23.3% caused by premature commitment. Our work provides both a controlled evaluation framework and concrete recommendations for improving the reliability of LLMs in multi-turn medical diagnosis.