Pre-training Transformers for Knowledge Graph Completion

TL;DR

This paper introduces iHT, a Transformer-based pre-training model for knowledge graph completion that achieves state-of-the-art results and demonstrates transferability across different KGs.

Contribution

The paper proposes a novel inductive KG representation model using large-scale pre-training with Transformers, improving performance and transferability over prior methods.

Findings

Achieves over 25% relative improvement in mean reciprocal rank on Wikidata5M.

Outperforms previous SOTA models on KG completion benchmarks.

Pre-trained iHT representations transfer effectively to smaller KGs with entity and relational shifts.

Abstract

Learning transferable representation of knowledge graphs (KGs) is challenging due to the heterogeneous, multi-relational nature of graph structures. Inspired by Transformer-based pretrained language models' success on learning transferable representation for texts, we introduce a novel inductive KG representation model (iHT) for KG completion by large-scale pre-training. iHT consists of a entity encoder (e.g., BERT) and a neighbor-aware relational scoring function both parameterized by Transformers. We first pre-train iHT on a large KG dataset, Wikidata5M. Our approach achieves new state-of-the-art results on matched evaluations, with a relative improvement of more than 25% in mean reciprocal rank over previous SOTA models. When further fine-tuned on smaller KGs with either entity and relational shifts, pre-trained iHT representations are shown to be transferable, significantly improving the performance on FB15K-237 and WN18RR.