VeriSim: A Configurable Framework for Evaluating Medical AI Under Realistic Patient Noise

TL;DR

VeriSim is an open-source framework that injects realistic patient communication noise into medical AI evaluations, revealing significant model performance degradation under authentic clinical conditions.

Contribution

We introduce VeriSim, a novel patient simulation framework that systematically incorporates clinically grounded noise into medical AI assessments, highlighting robustness gaps.

Findings

All models' diagnostic accuracy drops 15-25% under realistic noise.

Smaller models (7B) degrade 40% more than larger models (70B+).

Medical fine-tuning offers limited robustness improvements.

Abstract

Medical large language models (LLMs) achieve impressive performance on standardized benchmarks, yet these evaluations fail to capture the complexity of real clinical encounters where patients exhibit memory gaps, limited health literacy, anxiety, and other communication barriers. We introduce VeriSim, a truth-preserving patient simulation framework that injects controllable, clinically evidence-grounded noise into patient responses while maintaining strict adherence to medical ground truth through a hybrid UMLS-LLM verification mechanism. Our framework operationalizes six noise dimensions derived from peer-reviewed medical communication literature, capturing authentic clinical phenomena such as patient recall limitations, health literacy barriers, and stigma-driven non-disclosure. Experiments across seven open-weight LLMs reveal that all models degrade significantly under realistic patient noise, with diagnostic accuracy dropping 15-25% and conversation length increasing 34-55%. Notably, smaller models (7B) show 40% greater degradation than larger models (70B+), while medical fine-tuning on standard corpora provides limited robustness benefits against patient communication noise. Evaluation by board-certified clinicians demonstrates high-quality simulation with strong inter-annotator agreement (kappa > 0.80), while LLM-as-a-Judge serves as a validated auxiliary evaluator achieving comparable reliability for scalable assessment. Our results highlight a critical Sim-to-Real gap in current medical AI. We release VeriSim as an open-source noise-injection framework, establishing a rigorous testbed for evaluating clinical robustness.