Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes

TL;DR

This study evaluates how lab test results influence the accuracy of large language models in generating differential diagnoses from clinical vignettes, highlighting GPT-4's superior performance and the importance of lab data.

Contribution

It systematically assesses the impact of lab results on LLMs' diagnostic accuracy, comparing multiple models and incorporating clinician and knowledge graph evaluations.

Findings

GPT-4 achieved 55% accuracy for Top 1 diagnoses with lab data.

Lab results significantly improved diagnostic accuracy across models.

LLMs correctly interpreted common lab tests for differential diagnosis.

Abstract

Differential diagnosis is crucial for medicine as it helps healthcare providers systematically distinguish between conditions that share similar symptoms. This study assesses the impact of lab test results on differential diagnoses (DDx) made by large language models (LLMs). Clinical vignettes from 50 case reports from PubMed Central were created incorporating patient demographics, symptoms, and lab results. Five LLMs GPT-4, GPT-3.5, Llama-2-70b, Claude-2, and Mixtral-8x7B were tested to generate Top 10, Top 5, and Top 1 DDx with and without lab data. A comprehensive evaluation involving GPT-4, a knowledge graph, and clinicians was conducted. GPT-4 performed best, achieving 55% accuracy for Top 1 diagnoses and 60% for Top 10 with lab data, with lenient accuracy up to 80%. Lab results significantly improved accuracy, with GPT-4 and Mixtral excelling, though exact match rates were low. Lab tests, including liver function, metabolic/toxicology panels, and serology/immune tests, were generally interpreted correctly by LLMs for differential diagnosis.