Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT

TL;DR

This paper introduces Mixture-of-Partitions (MoP), a method for infusing large biomedical knowledge graphs into BERT models by partitioning the graph and fine-tuning lightweight adapters, leading to improved task performance.

Contribution

The paper presents a novel approach, MoP, that effectively incorporates large biomedical knowledge graphs into BERT using partitioning and adapters, enhancing performance on multiple tasks.

Findings

MoP improves BERT performance across six biomedical tasks.

Achieves new state-of-the-art results on five datasets.

Effectively handles very large knowledge graphs with partitioning.

Abstract

Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. In this paper, we proposed Mixture-of-Partitions (MoP), an infusion approach that can handle a very large knowledge graph (KG) by partitioning it into smaller sub-graphs and infusing their specific knowledge into various BERT models using lightweight adapters. To leverage the overall factual knowledge for a target task, these sub-graph adapters are further fine-tuned along with the underlying BERT through a mixture layer. We evaluate our MoP with three biomedical BERTs (SciBERT, BioBERT, PubmedBERT) on six downstream tasks (inc. NLI, QA, Classification), and the results show that our MoP consistently enhances the underlying BERTs in task performance, and achieves new SOTA performances on five evaluated datasets.