Towards Democratization of Subspeciality Medical Expertise

TL;DR

This study evaluates AMIE, a large language model, for supporting cardiology diagnosis and management, demonstrating it can enhance general cardiologists' performance in complex cases and potentially democratize subspecialty expertise.

Contribution

Introduces AMIE, an LLM-based system, and demonstrates its effectiveness in augmenting cardiology decision-making in complex cases through a real-world dataset and structured evaluation.

Findings

AMIE outperformed general cardiologists in 5 of 10 diagnostic domains.

Access to AMIE improved cardiologists' overall response quality in 63.7% of cases.

AMIE responses were superior across all 10 evaluated domains.

Abstract

The scarcity of subspecialist medical expertise, particularly in rare, complex and life-threatening diseases, poses a significant challenge for healthcare delivery. This issue is particularly acute in cardiology where timely, accurate management determines outcomes. We explored the potential of AMIE (Articulate Medical Intelligence Explorer), a large language model (LLM)-based experimental AI system optimized for diagnostic dialogue, to potentially augment and support clinical decision-making in this challenging context. We curated a real-world dataset of 204 complex cases from a subspecialist cardiology practice, including results for electrocardiograms, echocardiograms, cardiac MRI, genetic tests, and cardiopulmonary stress tests. We developed a ten-domain evaluation rubric used by subspecialists to evaluate the quality of diagnosis and clinical management plans produced by general cardiologists or AMIE, the latter enhanced with web-search and self-critique capabilities. AMIE was rated superior to general cardiologists for 5 of the 10 domains (with preference ranging from 9% to 20%), and equivalent for the rest. Access to AMIE's response improved cardiologists' overall response quality in 63.7% of cases while lowering quality in just 3.4%. Cardiologists' responses with access to AMIE were superior to cardiologist responses without access to AMIE for all 10 domains. Qualitative examinations suggest AMIE and general cardiologist could complement each other, with AMIE thorough and sensitive, while general cardiologist concise and specific. Overall, our results suggest that specialized medical LLMs have the potential to augment general cardiologists' capabilities by bridging gaps in subspecialty expertise, though further research and validation are essential for wide clinical utility.