Injecting Knowledge into Biomedical Pre-trained Models via Polymorphism and Synonymous Substitution

TL;DR

This paper introduces a simple method to enhance biomedical pre-trained language models by injecting relational knowledge through entity switching, improving their understanding and performance on biomedical tasks.

Contribution

The paper proposes a novel approach to inject relational knowledge into biomedical PLMs using polymorphism and synonymous substitution, addressing report bias issues.

Findings

Improved relational knowledge capture in PLMs

Enhanced performance on biomedical downstream tasks

Effective knowledge injection method

Abstract

Pre-trained language models (PLMs) were considered to be able to store relational knowledge present in the training data. However, some relational knowledge seems to be discarded unsafely in PLMs due to \textbf{report bias}: low-frequency relational knowledge might be underexpressed compared to high-frequency one in PLMs. This gives us a hint that relational knowledge might not be redundant to the stored knowledge of PLMs, but rather be complementary. To additionally inject relational knowledge into PLMs, we propose a simple-yet-effective approach to inject relational knowledge into PLMs, which is inspired by three observations (namely, polymorphism, synonymous substitution, and association). In particular, we switch entities in the training corpus to related entities (either hypernyms/hyponyms/synonyms, or arbitrarily-related concepts). Experimental results show that the proposed approach could not only better capture relational knowledge, but also improve the performance in various biomedical downstream tasks. Our model is available in \url{https://github.com/StevenZHB/BioPLM_InjectingKnowledge}.