HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

TL;DR

HealthBench Professional is an open benchmark that evaluates large language models on real clinician tasks, including care consults, documentation, and research, using physician-scored examples to measure progress.

Contribution

It introduces a new, carefully curated benchmark with real-world clinical examples and scoring rubrics to assess LLM performance in healthcare settings.

Findings

GPT-5.4 in ChatGPT for Clinicians outperforms other models and human physicians.

Benchmark includes difficult and adversarial examples to challenge models.

Human physician responses serve as a high-quality baseline.

Abstract

Millions of clinicians use ChatGPT to support clinical care, but evaluations of the most common use cases in model-clinician conversations are limited. We introduce HealthBench Professional, an open benchmark for evaluating large language models on real tasks that clinicians bring to ChatGPT in the course of their work. The benchmark is organized around three common use cases central to clinical practice: care consult, writing and documentation, and medical research. Each example includes a physician-authored conversation with ChatGPT for Clinicians and is scored via rubrics written and iteratively adjudicated by three or more physicians across three phases. HealthBench Professional examples were carefully selected for quality, representativeness, and difficulty for OpenAI's current frontier models, to enable continued measurement of progress. Difficult examples for recent OpenAI models were enriched by roughly 3.5 times relative to the candidate pool of 15,079 examples. Additionally, about one-third of examples involve physicians conducting deliberate adversarial testing of models. As a strong baseline, we also collected human physician responses for all tasks (unbounded time, specialist-matched, web access). The best scoring system, GPT-5.4 in ChatGPT for Clinicians, outperforms base GPT-5.4, all other models, and human physicians. We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care.