ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning

TL;DR

ClinDEF introduces a dynamic, multi-turn evaluation framework for large language models in clinical reasoning, simulating diagnostic dialogues to reveal reasoning gaps beyond static accuracy measures.

Contribution

This work presents ClinDEF, a novel dynamic evaluation framework grounded in disease knowledge graphs, enabling nuanced assessment of LLMs in clinical reasoning through simulated dialogues.

Findings

ClinDEF exposes critical reasoning gaps in current LLMs.

The framework provides fine-grained efficiency and diagnostic quality metrics.

Experiments demonstrate ClinDEF's effectiveness over static benchmarks.

Abstract

Clinical diagnosis begins with doctor-patient interaction, during which physicians iteratively gather information, determine examination and refine differential diagnosis through patients' response. This dynamic clinical-reasoning process is poorly represented by existing LLM benchmarks that focus on static question-answering. To mitigate these gaps, recent methods explore dynamic medical frameworks involving interactive clinical dialogues. Although effective, they often rely on limited, contamination-prone datasets and lack granular, multi-level evaluation. In this work, we propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues. Grounded in a disease knowledge graph, our method dynamically generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent. Our evaluation protocol goes beyond diagnostic accuracy by incorporating fine-grained efficiency analysis and rubric-based assessment of diagnostic quality. Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs, offering a more nuanced and clinically meaningful evaluation paradigm.