Using Graph Algorithms to Pretrain Graph Completion Transformers

TL;DR

This paper explores various graph algorithm-based pretraining tasks for Transformer models to improve knowledge graph completion, introducing a new path-finding algorithm that enhances performance across multiple datasets.

Contribution

It introduces a novel path-finding pretraining algorithm guided by information gain and demonstrates its effectiveness in knowledge graph completion tasks.

Findings

Path-finding pretraining improves MRR by 2-3%.

Combining all pretraining signals yields the best results.

Our method surpasses existing knowledge graph embedding methods on multiple datasets.

Abstract

Recent work on Graph Neural Networks has demonstrated that self-supervised pretraining can further enhance performance on downstream graph, link, and node classification tasks. However, the efficacy of pretraining tasks has not been fully investigated for downstream large knowledge graph completion tasks. Using a contextualized knowledge graph embedding approach, we investigate five different pretraining signals, constructed using several graph algorithms and no external data, as well as their combination. We leverage the versatility of our Transformer-based model to explore graph structure generation pretraining tasks (i.e. path and k-hop neighborhood generation), typically inapplicable to most graph embedding methods. We further propose a new path-finding algorithm guided by information gain and find that it is the best-performing pretraining task across three downstream knowledge graph completion datasets. While using our new path-finding algorithm as a pretraining signal provides 2-3% MRR improvements, we show that pretraining on all signals together gives the best knowledge graph completion results. In a multitask setting that combines all pretraining tasks, our method surpasses the latest and strong performing knowledge graph embedding methods on all metrics for FB15K-237, on MRR and Hit@1 for WN18RRand on MRR and hit@10 for JF17K (a knowledge hypergraph dataset).