Improving Clinical Documentation with AI: A Comparative Study of Sporo AI Scribe and GPT-4o mini

TL;DR

This study compares Sporo AI Scribe and GPT-4o Mini in clinical documentation, showing Sporo's superior accuracy, clinician satisfaction, and reliability in real-world healthcare settings.

Contribution

It introduces a multi-agent system leveraging fine-tuned medical LLMs and provides a comparative evaluation against GPT-4o Mini in clinical documentation tasks.

Findings

Sporo AI outperforms GPT-4o Mini in recall, precision, and F1 scores.

Clinicians rated Sporo summaries higher for accuracy and relevance.

Sporo AI demonstrates effective, reliable documentation with fewer hallucinations.

Abstract

AI-powered medical scribes have emerged as a promising solution to alleviate the documentation burden in healthcare. Ambient AI scribes provide real-time transcription and automated data entry into Electronic Health Records (EHRs), with the potential to improve efficiency, reduce costs, and enhance scalability. Despite early success, the accuracy of AI scribes remains critical, as errors can lead to significant clinical consequences. Additionally, AI scribes face challenges in handling the complexity and variability of medical language and ensuring the privacy of sensitive patient data. This case study aims to evaluate Sporo Health's AI scribe, a multi-agent system leveraging fine-tuned medical LLMs, by comparing its performance with OpenAI's GPT-4o Mini on multiple performance metrics. Using a dataset of de-identified patient conversation transcripts, AI-generated summaries were compared to clinician-generated notes (the ground truth) based on clinical content recall, precision, and F1 scores. Evaluations were further supplemented by clinician satisfaction assessments using a modified Physician Documentation Quality Instrument revision 9 (PDQI-9), rated by both a medical student and a physician. The results show that Sporo AI consistently outperformed GPT-4o Mini, achieving higher recall, precision, and overall F1 scores. Moreover, the AI generated summaries provided by Sporo were rated more favorably in terms of accuracy, comprehensiveness, and relevance, with fewer hallucinations. These findings demonstrate that Sporo AI Scribe is an effective and reliable tool for clinical documentation, enhancing clinician workflows while maintaining high standards of privacy and security.