Beyond the Basics: Leveraging Large Language Model for Fine-Grained Medical Entity Recognition

TL;DR

This paper evaluates the LLaMA3 model for fine-grained medical entity recognition in clinical texts, employing various learning paradigms and selection methods, achieving significant performance improvements over zero-shot and few-shot approaches.

Contribution

It introduces a comprehensive evaluation of LLaMA3 for detailed medical entity recognition using multiple learning strategies and novel example selection methods, ensuring fair comparison.

Findings

Fine-tuned LLaMA3 outperforms zero-shot and few-shot methods by over 60%.

Achieved an F1 score of 81.24% in detailed medical entity recognition.

Embedding-based example selection improves few-shot learning performance.

Abstract

Extracting clinically relevant information from unstructured medical narratives such as admission notes, discharge summaries, and emergency case histories remains a challenge in clinical natural language processing (NLP). Medical Entity Recognition (MER) identifies meaningful concepts embedded in these records. Recent advancements in large language models (LLMs) have shown competitive MER performance; however, evaluations often focus on general entity types, offering limited utility for real-world clinical needs requiring finer-grained extraction. To address this gap, we rigorously evaluated the open-source LLaMA3 model for fine-grained medical entity recognition across 18 clinically detailed categories. To optimize performance, we employed three learning paradigms: zero-shot, few-shot, and fine-tuning with Low-Rank Adaptation (LoRA). To further enhance few-shot learning, we introduced two example selection methods based on token- and sentence-level embedding similarity, utilizing a pre-trained BioBERT model. Unlike prior work assessing zero-shot and few-shot performance on proprietary models (e.g., GPT-4) or fine-tuning different architectures, we ensured methodological consistency by applying all strategies to a unified LLaMA3 backbone, enabling fair comparison across learning settings. Our results showed that fine-tuned LLaMA3 surpasses zero-shot and few-shot approaches by 63.11% and 35.63%, respectivel respectively, achieving an F1 score of 81.24% in granular medical entity extraction.