Dynamic Q&A of Clinical Documents with Large Language Models

TL;DR

This paper presents a natural language interface powered by large language models for dynamic question-answering on clinical notes, demonstrating promising accuracy and efficiency improvements, but highlighting challenges like hallucinations and limited evaluation scope.

Contribution

Introduces a LLM-based chatbot for clinical note querying, optimizing model performance and latency, and evaluating different embedding models and LLMs in a healthcare context.

Findings

Wizard Vicuna achieves superior accuracy

Model quantization reduces latency by 48 times

Challenges include hallucinations and limited case diversity

Abstract

Electronic health records (EHRs) house crucial patient data in clinical notes. As these notes grow in volume and complexity, manual extraction becomes challenging. This work introduces a natural language interface using large language models (LLMs) for dynamic question-answering on clinical notes. Our chatbot, powered by Langchain and transformer-based LLMs, allows users to query in natural language, receiving relevant answers from clinical notes. Experiments, utilizing various embedding models and advanced LLMs, show Wizard Vicuna's superior accuracy, albeit with high compute demands. Model optimization, including weight quantization, improves latency by approximately 48 times. Promising results indicate potential, yet challenges such as model hallucinations and limited diverse medical case evaluations remain. Addressing these gaps is crucial for unlocking the value in clinical notes and advancing AI-driven clinical decision-making.