Harnessing Large Language Models for Precision Querying and Retrieval-Augmented Knowledge Extraction in Clinical Data Science

TL;DR

This paper explores the use of Large Language Models for precise querying and information extraction from clinical electronic health records, demonstrating their potential in supporting clinical workflows through a new evaluation framework.

Contribution

It introduces a flexible evaluation framework for LLMs in clinical data tasks and assesses their performance on structured and unstructured EHR data using synthetic datasets.

Findings

LLMs can accurately query structured EHR data.

LLMs reliably extract information from clinical notes with RAG.

Evaluation metrics show promising results for clinical applications.

Abstract

This study applies Large Language Models (LLMs) to two foundational Electronic Health Record (EHR) data science tasks: structured data querying (using programmatic languages, Python/Pandas) and information extraction from unstructured clinical text via a Retrieval Augmented Generation (RAG) pipeline. We test the ability of LLMs to interact accurately with large structured datasets for analytics and the reliability of LLMs in extracting semantically correct information from free text health records when supported by RAG. To this end, we presented a flexible evaluation framework that automatically generates synthetic question and answer pairs tailored to the characteristics of each dataset or task. Experiments were conducted on a curated subset of MIMIC III, (four structured tables and one clinical note type), using a mix of locally hosted and API-based LLMs. Evaluation combined exact-match metrics, semantic similarity, and human judgment. Our findings demonstrate the potential of LLMs to support precise querying and accurate information extraction in clinical workflows.