On-the-fly Definition Augmentation of LLMs for Biomedical NER

TL;DR

This paper introduces a method for enhancing biomedical named entity recognition in large language models by dynamically incorporating relevant definitions, significantly improving performance especially in limited data scenarios.

Contribution

It proposes a novel on-the-fly definition augmentation technique for LLMs that improves biomedical NER accuracy and explores effective prompting strategies for knowledge integration.

Findings

Definition augmentation yields a 15% relative F1 score improvement in GPT-4.

Careful prompting strategies enhance LLM performance beyond fine-tuned models.

The approach is effective for both open source and closed LLMs.

Abstract

Despite their general capabilities, LLMs still struggle on biomedical NER tasks, which are difficult due to the presence of specialized terminology and lack of training data. In this work we set out to improve LLM performance on biomedical NER in limited data settings via a new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly. During this process, to provide a test bed for knowledge augmentation, we perform a comprehensive exploration of prompting strategies. Our experiments show that definition augmentation is useful for both open source and closed LLMs. For example, it leads to a relative improvement of 15\% (on average) in GPT-4 performance (F1) across all (six) of our test datasets. We conduct extensive ablations and analyses to demonstrate that our performance improvements stem from adding relevant definitional knowledge. We find that careful prompting strategies also improve LLM performance, allowing them to outperform fine-tuned language models in few-shot settings. To facilitate future research in this direction, we release our code at https://github.com/allenai/beacon.