A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI

TL;DR

This paper presents a scalable benchmarking framework for evaluating health AI chatbots' diagnostic accuracy using clinical vignettes, demonstrating its application with an AI chatbot that outperforms traditional symptom checkers.

Contribution

The study introduces a reproducible, scalable benchmarking methodology for health AI systems and applies it to evaluate an AI chatbot's diagnostic performance across multiple specialties.

Findings

AI chatbot achieved 81.8% top-one accuracy

System required 47% fewer questions than traditional symptom checkers

Chatbot demonstrated 95.8% accuracy in specialist referrals

Abstract

Diagnostic errors in healthcare persist as a critical challenge, with increasing numbers of patients turning to online resources for health information. While AI-powered healthcare chatbots show promise, there exists no standardized and scalable framework for evaluating their diagnostic capabilities. This study introduces a scalable benchmarking methodology for assessing health AI systems and demonstrates its application through August, an AI-driven conversational chatbot. Our methodology employs 400 validated clinical vignettes across 14 medical specialties, using AI-powered patient actors to simulate realistic clinical interactions. In systematic testing, August achieved a top-one diagnostic accuracy of 81.8% (327/400 cases) and a top-two accuracy of 85.0% (340/400 cases), significantly outperforming traditional symptom checkers. The system demonstrated 95.8% accuracy in specialist referrals and required 47% fewer questions compared to conventional symptom checkers (mean 16 vs 29 questions), while maintaining empathetic dialogue throughout consultations. These findings demonstrate the potential of AI chatbots to enhance healthcare delivery, though implementation challenges remain regarding real-world validation and integration of objective clinical data. This research provides a reproducible framework for evaluating healthcare AI systems, contributing to the responsible development and deployment of AI in clinical settings.