Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG

TL;DR

This paper compares continual pretraining and graph retrieval-augmented generation for injecting structured biomedical knowledge into language models, demonstrating improved performance on biomedical tasks and question answering benchmarks.

Contribution

It introduces a large-scale UMLS-based knowledge graph, constructs models with continual pretraining, and develops a GraphRAG pipeline for knowledge retrieval without retraining.

Findings

BERTUMLS outperforms BERT on knowledge-intensive tasks.

GraphRAG improves QA accuracy by over 3-5 points without retraining.

Effects vary depending on the base model's existing knowledge.

Abstract

The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two complementary strategies for leveraging structured knowledge from the UMLS Metathesaurus: (i) Continual pretraining that embeds knowledge into model parameters, and (ii) Graph Retrieval-Augmented Generation (GraphRAG) that consults a knowledge graph at inference time. We first construct a large-scale biomedical knowledge graph from UMLS (3.4 million concepts and 34.2 million relations), stored in Neo4j for efficient querying. We then derive a ~100-million-token textual corpus from this graph to continually pretrain two models: BERTUMLS (from BERT) and BioBERTUMLS (from BioBERT). We evaluate these models on six BLURB (Biomedical Language Understanding and Reasoning Benchmark) datasets spanning five task types and evaluate GraphRAG on the two QA (Question Answering) datasets (PubMedQA, BioASQ). On BLURB tasks, BERTUMLS improves over BERT, with the largest gains on knowledge-intensive QA. Effects on BioBERT are more nuanced, suggesting diminishing returns when the base model already encodes substantial biomedical text knowledge. Finally, augmenting LLaMA 3-8B with our GraphRAG pipeline yields over than 3 points accuracy on PubMedQA and 5 points on BioASQ without any retraining, delivering transparent, multi-hop, and easily updated knowledge access. We release the processed UMLS Neo4j graph to support reproducibility.