Identifying and Extracting Rare Disease Phenotypes with Large Language Models

TL;DR

This study evaluates ChatGPT's ability to extract rare disease phenotypes from unstructured text in zero- and few-shot settings, comparing it to traditional fine-tuning methods, and establishes a benchmark for this task.

Contribution

It introduces novel prompts for ChatGPT to extract rare disease phenotypes and benchmarks its performance against fine-tuned models, highlighting potential and limitations.

Findings

Fine-tuning BioClinicalBERT outperforms ChatGPT overall (F1 0.689 vs. 0.472/0.591).

ChatGPT matches or exceeds fine-tuned models for certain entities in one-shot settings.

Prompt engineering can enhance ChatGPT's entity extraction performance with minimal labeled data.

Abstract

Rare diseases (RDs) are collectively common and affect 300 million people worldwide. Accurate phenotyping is critical for informing diagnosis and treatment, but RD phenotypes are often embedded in unstructured text and time-consuming to extract manually. While natural language processing (NLP) models can perform named entity recognition (NER) to automate extraction, a major bottleneck is the development of a large, annotated corpus for model training. Recently, prompt learning emerged as an NLP paradigm that can lead to more generalizable results without any (zero-shot) or few labeled samples (few-shot). Despite growing interest in ChatGPT, a revolutionary large language model capable of following complex human prompts and generating high-quality responses, none have studied its NER performance for RDs in the zero- and few-shot settings. To this end, we engineered novel prompts aimed at extracting RD phenotypes and, to the best of our knowledge, are the first the establish a benchmark for evaluating ChatGPT's performance in these settings. We compared its performance to the traditional fine-tuning approach and conducted an in-depth error analysis. Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.591 in the zero- and few-shot settings, respectively). Despite this, ChatGPT achieved similar or higher accuracy for certain entities (i.e., rare diseases and signs) in the one-shot setting (F1 of 0.776 and 0.725). This suggests that with appropriate prompt engineering, ChatGPT has the potential to match or outperform fine-tuned language models for certain entity types with just one labeled sample. While the proliferation of large language models may provide opportunities for supporting RD diagnosis and treatment, researchers and clinicians should critically evaluate model outputs and be well-informed of their limitations.