MedPI: Evaluating AI Systems in Medical Patient-facing Interactions

TL;DR

MedPI introduces a comprehensive, multi-dimensional benchmark for evaluating large language models in medical patient-clinician dialogues, assessing safety, communication, and diagnostic capabilities across 105 criteria.

Contribution

This work presents MedPI, a novel high-dimensional benchmark with a detailed evaluation framework and calibrated AI judges for assessing LLMs in complex medical conversations.

Findings

All evaluated LLMs show low performance on diagnostic and communication dimensions.

The benchmark reveals significant gaps in current models' medical understanding.

MedPI can guide future development of safer, more effective medical AI systems.

Abstract

We present MedPI, a high-dimensional benchmark for evaluating large language models (LLMs) in patient-clinician conversations. Unlike single-turn question-answer (QA) benchmarks, MedPI evaluates the medical dialogue across 105 dimensions comprising the medical process, treatment safety, treatment outcomes and doctor-patient communication across a granular, accreditation-aligned rubric. MedPI comprises five layers: (1) Patient Packets (synthetic EHR-like ground truth); (2) an AI Patient instantiated through an LLM with memory and affect; (3) a Task Matrix spanning encounter reasons (e.g. anxiety, pregnancy, wellness checkup) x encounter objectives (e.g. diagnosis, lifestyle advice, medication advice); (4) an Evaluation Framework with 105 dimensions on a 1-4 scale mapped to the Accreditation Council for Graduate Medical Education (ACGME) competencies; and (5) AI Judges that are calibrated, committee-based LLMs providing scores, flags, and evidence-linked rationales. We evaluate 9 flagship models -- Claude Opus 4.1, Claude Sonnet 4, MedGemma, Gemini 2.5 Pro, Llama 3.3 70b Instruct, GPT-5, GPT OSS 120b, o3, Grok-4 -- across 366 AI Patients and 7,097 conversations using a standardized "vanilla clinician" prompt. For all LLMs, we observe low performance across a variety of dimensions, in particular on differential diagnosis. Our work can help guide future use of LLMs for diagnosis and treatment recommendations.