Scalable Medication Extraction and Discontinuation Identification from Electronic Health Records Using Large Language Models

TL;DR

This study evaluates large language models for extracting medication information and identifying discontinuations from EHR notes, demonstrating high performance and scalability, especially with GPT-4o and open-source models, without extensive human annotation.

Contribution

It systematically compares multiple LLMs for medication extraction and discontinuation classification from EHRs, highlighting the potential of open-source models as scalable alternatives.

Findings

GPT-4o achieved highest F1 scores: 94.0% extraction, 78.1% classification.

Open-source Llama-3.1-70B-Instruct performed well in classification tasks.

Few-shot learning improved model performance, CoT reasoning had mixed effects.

Abstract

Identifying medication discontinuations in electronic health records (EHRs) is vital for patient safety but is often hindered by information being buried in unstructured notes. This study aims to evaluate the capabilities of advanced open-sourced and proprietary large language models (LLMs) in extracting medications and classifying their medication status from EHR notes, focusing on their scalability on medication information extraction without human annotation. We collected three EHR datasets from diverse sources to build the evaluation benchmark. We evaluated 12 advanced LLMs and explored multiple LLM prompting strategies. Performance on medication extraction, medication status classification, and their joint task (extraction then classification) was systematically compared across all experiments. We found that LLMs showed promising performance on the medication extraction and discontinuation classification from EHR notes. GPT-4o consistently achieved the highest average F1 scores in all tasks under zero-shot setting - 94.0% for medication extraction, 78.1% for discontinuation classification, and 72.7% for the joint task. Open-sourced models followed closely, Llama-3.1-70B-Instruct achieved the highest performance in medication status classification on the MIV-Med dataset (68.7%) and in the joint task on both the Re-CASI (76.2%) and MIV-Med (60.2%) datasets. Medical-specific LLMs demonstrated lower performance compared to advanced general-domain LLMs. Few-shot learning generally improved performance, while CoT reasoning showed inconsistent gains. LLMs demonstrate strong potential for medication extraction and discontinuation identification on EHR notes, with open-sourced models offering scalable alternatives to proprietary systems and few-shot can further improve LLMs' capability.