oRetrieval Augmented Generation for 10 Large Language Models and its Generalizability in Assessing Medical Fitness

TL;DR

This study evaluates retrieval-augmented large language models for medical preoperative assessments, demonstrating high accuracy, speed, and consistency across guidelines, with GPT4 achieving 96.4% correctness and no hallucinations.

Contribution

It introduces LLM-RAG models tailored for medical preoperative tasks, showing their effectiveness and generalizability across diverse clinical guidelines.

Findings

GPT4 LLM-RAG achieved 96.4% accuracy in assessments.

Models responded within 20 seconds, faster than clinicians.

Responses were consistent and hallucination-free.

Abstract

Large Language Models (LLMs) show potential for medical applications but often lack specialized clinical knowledge. Retrieval Augmented Generation (RAG) allows customization with domain-specific information, making it suitable for healthcare. This study evaluates the accuracy, consistency, and safety of RAG models in determining fitness for surgery and providing preoperative instructions. We developed LLM-RAG models using 35 local and 23 international preoperative guidelines and tested them against human-generated responses. A total of 3,682 responses were evaluated. Clinical documents were processed using Llamaindex, and 10 LLMs, including GPT3.5, GPT4, and Claude-3, were assessed. Fourteen clinical scenarios were analyzed, focusing on seven aspects of preoperative instructions. Established guidelines and expert judgment were used to determine correct responses, with human-generated answers serving as comparisons. The LLM-RAG models generated responses within 20 seconds, significantly faster than clinicians (10 minutes). The GPT4 LLM-RAG model achieved the highest accuracy (96.4% vs. 86.6%, p=0.016), with no hallucinations and producing correct instructions comparable to clinicians. Results were consistent across both local and international guidelines. This study demonstrates the potential of LLM-RAG models for preoperative healthcare tasks, highlighting their efficiency, scalability, and reliability.