Classification performance and reproducibility of GPT-4 omni for information extraction from veterinary electronic health records

TL;DR

This study evaluates GPT-4 omni's ability to extract clinical signs from veterinary EHRs, demonstrating high accuracy and reproducibility, outperforming GPT-3.5 Turbo and showing robustness across different settings.

Contribution

It provides a comprehensive comparison of GPT-4 omni and GPT-3.5 Turbo for veterinary EHR extraction, highlighting GPT-4 omni's superior performance and stability regardless of temperature adjustments.

Findings

GPT-4 omni achieved 96.9% sensitivity and 97.6% specificity.

GPT-4 omni outperformed GPT-3.5 Turbo, especially in sensitivity.

Reproducibility of GPT-4 omni was higher than human interobserver agreement.

Abstract

Large language models (LLMs) can extract information from veterinary electronic health records (EHRs), but performance differences between models, the effect of temperature settings, and the influence of text ambiguity have not been previously evaluated. This study addresses these gaps by comparing the performance of GPT-4 omni (GPT-4o) and GPT-3.5 Turbo under different conditions and investigating the relationship between human interobserver agreement and LLM errors. The LLMs and five humans were tasked with identifying six clinical signs associated with Feline chronic enteropathy in 250 EHRs from a veterinary referral hospital. At temperature 0, the performance of GPT-4o compared to the majority opinion of human respondents, achieved 96.9% sensitivity (interquartile range [IQR] 92.9-99.3%), 97.6% specificity (IQR 96.5-98.5%), 80.7% positive predictive value (IQR 70.8-84.6%), 99.5% negative predictive value (IQR 99.0-99.9%), 84.4% F1 score (IQR 77.3-90.4%), and 96.3% balanced accuracy (IQR 95.0-97.9%). The performance of GPT-4o was significantly better than that of its predecessor, GPT-3.5 Turbo, particularly with respect to sensitivity where GPT-3.5 Turbo only achieved 81.7% (IQR 78.9-84.8%). Adjusting the temperature for GPT-4o did not significantly impact classification performance. GPT-4o demonstrated greater reproducibility than human pairs regardless of temperature, with an average Cohen's kappa of 0.98 (IQR 0.98-0.99) at temperature 0 compared to 0.8 (IQR 0.78-0.81) for humans. Most GPT-4o errors occurred in instances where humans disagreed (35/43 errors, 81.4%), suggesting that these errors were more likely caused by ambiguity of the EHR than explicit model faults. Using GPT-4o to automate information extraction from veterinary EHRs is a viable alternative to manual extraction.