Mapping Biomedical Ontology Terms to IDs: Effect of Domain Prevalence on Prediction Accuracy

TL;DR

This study investigates how the prevalence of biomedical ontology IDs in literature influences the accuracy of large language models in mapping terms to IDs, revealing high accuracy for high-prevalence IDs but limitations for low-prevalence ones.

Contribution

It demonstrates that ontology ID prevalence in biomedical literature significantly affects LLM mapping accuracy and highlights the lexicalization of high-prevalence gene symbols, informing future biomedical NLP model development.

Findings

High prevalence predicts better mapping accuracy for HPO, GO, and UniProtKB IDs.

GPT-4 achieves 95% accuracy in mapping protein names to HUGO gene symbols.

Mapping accuracy for HUGO gene symbols is unaffected by prevalence due to lexicalization.

Abstract

This study evaluates the ability of large language models (LLMs) to map biomedical ontology terms to their corresponding ontology IDs across the Human Phenotype Ontology (HPO), Gene Ontology (GO), and UniProtKB terminologies. Using counts of ontology IDs in the PubMed Central (PMC) dataset as a surrogate for their prevalence in the biomedical literature, we examined the relationship between ontology ID prevalence and mapping accuracy. Results indicate that ontology ID prevalence strongly predicts accurate mapping of HPO terms to HPO IDs, GO terms to GO IDs, and protein names to UniProtKB accession numbers. Higher prevalence of ontology IDs in the biomedical literature correlated with higher mapping accuracy. Predictive models based on receiver operating characteristic (ROC) curves confirmed this relationship. In contrast, this pattern did not apply to mapping protein names to Human Genome Organisation's (HUGO) gene symbols. GPT-4 achieved a high baseline performance (95%) in mapping protein names to HUGO gene symbols, with mapping accuracy unaffected by prevalence. We propose that the high prevalence of HUGO gene symbols in the literature has caused these symbols to become lexicalized, enabling GPT-4 to map protein names to HUGO gene symbols with high accuracy. These findings highlight the limitations of LLMs in mapping ontology terms to low-prevalence ontology IDs and underscore the importance of incorporating ontology ID prevalence into the training and evaluation of LLMs for biomedical applications.