Sem4SAP: Synonymous Expression Mining From Open Knowledge Graph For Language Model Synonym-Aware Pretraining

TL;DR

Sem4SAP introduces a framework that mines synsets from Open Knowledge Graphs and employs novel pretraining methods to enhance language models' understanding of synonyms, improving performance across multiple tasks.

Contribution

It is the first to leverage Open Knowledge Graphs for synset mining and integrate this into pretraining, significantly boosting PLMs' synonym comprehension.

Findings

Outperforms baseline models on ten tasks

Effectively injects synonym knowledge into PLMs

Enhances robustness to synonym substitution attacks

Abstract

The model's ability to understand synonymous expression is crucial in many kinds of downstream tasks. It will make the model to better understand the similarity between context, and more robust to the synonym substitution attack. However, many Pretrained Language Model (PLM) lack synonym knowledge due to limitation of small-scale synsets and PLM's pretraining objectives. In this paper, we propose a framework called Sem4SAP to mine synsets from Open Knowledge Graph (Open-KG) and using the mined synsets to do synonym-aware pretraining for language models. We propose to coarsly filter the content in Open-KG and use the frequency information to better help the clustering process under low-resource unsupervised conditions. We expand the mined synsets by migrating core semantics between synonymous expressions.We also propose two novel and effective synonym-aware pre-training methods for injecting synonym knowledge into PLMs.Extensive experiments demonstrate that Sem4SAP can dramatically outperform the original PLMs and other baselines on ten different tasks.