Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs

TL;DR

This paper introduces a lightweight adapter-based method to incorporate structured biomedical knowledge from knowledge graphs into pre-trained language models, enhancing performance on various biomedical NLP tasks with low computational costs.

Contribution

It proposes a novel approach using adapter modules and knowledge graph partitioning to inject structured biomedical knowledge into PLMs, improving task performance efficiently.

Findings

Performance improvements on multiple biomedical NLP tasks

Effective knowledge integration with low computational overhead

Insights into knowledge graph partitioning and adapter fine-tuning

Abstract

Recent advances in natural language processing (NLP) owe their success to pre-training language models on large amounts of unstructured data. Still, there is an increasing effort to combine the unstructured nature of LMs with structured knowledge and reasoning. Particularly in the rapidly evolving field of biomedical NLP, knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large language models and domain-specific knowledge, considering the available biomedical knowledge graphs (KGs) curated by experts over the decades. In this paper, we develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models (PLMs). We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical ontology OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. The approach includes partitioning knowledge graphs into smaller subgraphs, fine-tuning adapter modules for each subgraph, and combining the knowledge in a fusion layer. We test the performance on three downstream tasks: document classification,question answering, and natural language inference. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low. Finally, we provide a detailed interpretation of the results and report valuable insights for future work.