Human-AI Co-reasoning for Clinical Diagnosis with Evidence-Integrated Language Agent

TL;DR

PULSE is a medical reasoning AI that combines language models with literature retrieval to support clinical diagnosis, matching expert performance and aiding physicians across diverse cases, with insights into collaboration workflows and limitations.

Contribution

This work introduces PULSE, a novel AI agent integrating language models and scientific literature for clinical diagnosis, evaluated on a comprehensive endocrinology benchmark.

Findings

PULSE achieves expert-level accuracy in diagnosis.

It maintains stable performance across disease incidence levels.

Collaboration with PULSE improves physician diagnostic reasoning.

Abstract

We present PULSE, a medical reasoning agent that combines a domain-tuned large language model with scientific literature retrieval to support diagnostic decision-making in complex real-world cases. To evaluate its capabilities, we curated a benchmark of 82 authentic endocrinology case reports encompassing a broad spectrum of disease types and incidence levels. In controlled experiments, we compared PULSE's performance against physicians with varying levels of expertise-from residents to senior specialists-and examined how AI assistance influenced human diagnostic reasoning. PULSE attained expert-competitive accuracy, outperforming residents and junior specialists while matching senior specialist performance at both Top@1 and Top@4 thresholds. Unlike physicians, whose accuracy declined with disease rarity, PULSE maintained stable performance across incidence tiers. The agent also exhibited adaptive reasoning, increasing output length with case difficulty in a manner analogous to the longer deliberation observed among expert clinicians. When used collaboratively, PULSE enabled physicians to correct initial errors and broaden diagnostic hypotheses, but also introduced risks of automation bias. The study explores both serial and concurrent collaboration workflows, revealing that PULSE offers robust support across common and rare presentations. These findings underscore both the promise and the limitations of language model-based agents in clinical diagnosis, and offer a framework for evaluating their role in real-world decision-making.