MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

TL;DR

This paper introduces MedAction, a pipeline for training medical language models on multi-turn diagnostic trajectories, improving their ability to reason and act under evolving clinical evidence.

Contribution

We develop a novel distillation pipeline and dataset for active multi-turn clinical diagnosis, addressing current models' failure modes and enhancing open-source medical LLM performance.

Findings

Fine-tuning on MedAction-32K improves diagnostic reasoning accuracy.

MedAction-8B achieves state-of-the-art results on MedR-Bench.

Proposed metrics effectively filter high-quality diagnostic trajectories.

Abstract

Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplification of real clinical practice. We study active diagnosis: the real-life clinical process of starting from initial observation, ordering tests, interpreting results, and updating a differential diagnosis across multiple turns. Through systematic analysis, we identify three recurring failure modes in current LLMs: ungrounded test ordering, unreliable diagnostic update, and degraded multi-turn coherence. Together, these failures reveal a core deficit: existing medical training data teaches models to reason from complete information but not to act under evolving, partial evidence. To address this gap, we introduce MedAction, a tree-structured distillation pipeline that synthesizes diverse and high-quality multi-turn diagnostic trajectories via LLM-environment interaction. We propose two knowledge-graph-grounded metrics to filter trajectory quality: Disease Trajectory Consistency (DTC), which tracks whether the model's hypothesis converges toward the correct diagnosis, and Reasoning-Action Consistency (RAC), which verifies that belief updates are driven by gathered evidence. Using this pipeline, we construct MedAction-32K, a dataset of 32,681 trajectories from 2,896 PMC cases. Fine-tuning an 8B model on MedAction-32K achieves state-of-the-art performance among open-source models on both MedR-Bench and our curated MedAction-300-Hard benchmark, pushing the edge for open-source medical LLMs.