Enhancing Vaccine Safety Surveillance: Extracting Vaccine Mentions from Emergency Department Triage Notes Using Fine-Tuned Large Language Models

TL;DR

This paper demonstrates that fine-tuned large language models, specifically Llama 3, can effectively extract vaccine mentions from emergency department triage notes, enhancing real-time vaccine safety monitoring.

Contribution

It introduces a novel approach using fine-tuned Llama 3 models combined with prompt engineering for vaccine mention extraction from clinical notes.

Findings

Fine-tuned Llama 3 outperforms other models in accuracy.

Model quantization enables deployment in resource-limited settings.

Automates data extraction for improved vaccine safety surveillance.

Abstract

This study evaluates fine-tuned Llama 3.2 models for extracting vaccine-related information from emergency department triage notes to support near real-time vaccine safety surveillance. Prompt engineering was used to initially create a labeled dataset, which was then confirmed by human annotators. The performance of prompt-engineered models, fine-tuned models, and a rule-based approach was compared. The fine-tuned Llama 3 billion parameter model outperformed other models in its accuracy of extracting vaccine names. Model quantization enabled efficient deployment in resource-constrained environments. Findings demonstrate the potential of large language models in automating data extraction from emergency department notes, supporting efficient vaccine safety surveillance and early detection of emerging adverse events following immunization issues.