Generalizable and Scalable Multistage Biomedical Concept Normalization Leveraging Large Language Models

TL;DR

This study demonstrates that large language models, both proprietary and open-source, can significantly enhance biomedical concept normalization performance when integrated with existing rule-based systems without the need for fine-tuning.

Contribution

The paper introduces a two-step LLM integration approach that improves biomedical concept normalization by generating alternative phrasings and pruning candidate concepts, showing substantial performance gains.

Findings

GPT-3.5-turbo improves F1 by up to 10.9 points.

Open-source Vicuna model achieves up to 18.7 point increase in F1.

Large general-purpose LLMs can be effectively used without fine-tuning.

Abstract

Background: Biomedical entity normalization is critical to biomedical research because the richness of free-text clinical data, such as progress notes, can often be fully leveraged only after translating words and phrases into structured and coded representations suitable for analysis. Large Language Models (LLMs), in turn, have shown great potential and high performance in a variety of natural language processing (NLP) tasks, but their application for normalization remains understudied. Methods: We applied both proprietary and open-source LLMs in combination with several rule-based normalization systems commonly used in biomedical research. We used a two-step LLM integration approach, (1) using an LLM to generate alternative phrasings of a source utterance, and (2) to prune candidate UMLS concepts, using a variety of prompting methods. We measure results by $F_{\beta}$, where we favor recall over precision, and F1. Results: We evaluated a total of 5,523 concept terms and text contexts from a publicly available dataset of human-annotated biomedical abstracts. Incorporating GPT-3.5-turbo increased overall $F_{\beta}$ and F1 in normalization systems +9.5 and +7.3 (MetaMapLite), +13.9 and +10.9 (QuickUMLS), and +10.5 and +10.3 (BM25), while the open-source Vicuna model achieved +10.8 and +12.2 (MetaMapLite), +14.7 and +15 (QuickUMLS), and +15.6 and +18.7 (BM25). Conclusions: Existing general-purpose LLMs, both propriety and open-source, can be leveraged at scale to greatly improve normalization performance using existing tools, with no fine-tuning.