MoCoSA: Momentum Contrast for Knowledge Graph Completion with Structure-Augmented Pre-trained Language Models

TL;DR

MoCoSA enhances knowledge graph completion by integrating structure-aware pre-trained language models with momentum contrastive learning, significantly improving performance on benchmark datasets.

Contribution

This paper introduces MoCoSA, a novel method combining structure-augmented PLMs with momentum contrast for improved KGC, addressing limitations of existing structure-based and description-based methods.

Findings

Achieves state-of-the-art MRR on WN18RR and OpenBG500 datasets.

Outperforms existing methods with 2.5% and 21% improvements.

Demonstrates robustness in reasoning with unseen entities.

Abstract

Knowledge Graph Completion (KGC) aims to conduct reasoning on the facts within knowledge graphs and automatically infer missing links. Existing methods can mainly be categorized into structure-based or description-based. On the one hand, structure-based methods effectively represent relational facts in knowledge graphs using entity embeddings. However, they struggle with semantically rich real-world entities due to limited structural information and fail to generalize to unseen entities. On the other hand, description-based methods leverage pre-trained language models (PLMs) to understand textual information. They exhibit strong robustness towards unseen entities. However, they have difficulty with larger negative sampling and often lag behind structure-based methods. To address these issues, in this paper, we propose Momentum Contrast for knowledge graph completion with Structure-Augmented pre-trained language models (MoCoSA), which allows the PLM to perceive the structural information by the adaptable structure encoder. To improve learning efficiency, we proposed momentum hard negative and intra-relation negative sampling. Experimental results demonstrate that our approach achieves state-of-the-art performance in terms of mean reciprocal rank (MRR), with improvements of 2.5% on WN18RR and 21% on OpenBG500.