Retrieval augmented generation based dynamic prompting for few-shot biomedical named entity recognition using large language models

TL;DR

This paper proposes a retrieval-augmented dynamic prompting method for large language models to improve few-shot biomedical named entity recognition, demonstrating significant performance gains across multiple datasets.

Contribution

It introduces a novel retrieval-augmented dynamic prompting strategy that adaptively updates prompts based on input similarity, enhancing LLM performance in biomedical NER tasks.

Findings

Static prompting increases F1-scores by up to 12%.

Dynamic prompting with retrieval improves F1-scores by 5.6-7.3%.

Retrieval-based adaptive prompts significantly enhance few-shot biomedical NER.

Abstract

Biomedical named entity recognition (NER) is a high-utility natural language processing (NLP) task, and large language models (LLMs) show promise particularly in few-shot settings (i.e., limited training data). In this article, we address the performance challenges of LLMs for few-shot biomedical NER by investigating a dynamic prompting strategy involving retrieval-augmented generation (RAG). In our approach, the annotated in-context learning examples are selected based on their similarities with the input texts, and the prompt is dynamically updated for each instance during inference. We implemented and optimized static and dynamic prompt engineering techniques and evaluated them on five biomedical NER datasets. Static prompting with structured components increased average F1-scores by 12% for GPT-4, and 11% for GPT-3.5 and LLaMA 3-70B, relative to basic static prompting. Dynamic prompting further improved performance, with TF-IDF and SBERT retrieval methods yielding the best results, improving average F1-scores by 7.3% and 5.6% in 5-shot and 10-shot settings, respectively. These findings highlight the utility of contextually adaptive prompts via RAG for biomedical NER.