Global and local evaluation of link prediction tasks with neural embeddings

TL;DR

This paper introduces a unified benchmark for evaluating neural embeddings in knowledge graphs, comparing global versus local training methods for link prediction, and emphasizes transparency and reproducibility in evaluation practices.

Contribution

It formalizes a comprehensive evaluation methodology and empirically investigates the effectiveness of global training of neural embeddings for link prediction.

Findings

Global training of embeddings can outperform local training in certain scenarios.

The evaluation pipeline is open source, promoting transparency.

The benchmark facilitates consistent comparison across different methods.

Abstract

We focus our attention on the link prediction problem for knowledge graphs, which is treated herein as a binary classification task on neural embeddings of the entities. By comparing, combining and extending different methodologies for link prediction on graph-based data coming from different domains, we formalize a unified methodology for the quality evaluation benchmark of neural embeddings for knowledge graphs. This benchmark is then used to empirically investigate the potential of training neural embeddings globally for the entire graph, as opposed to the usual way of training embeddings locally for a specific relation. This new way of testing the quality of the embeddings evaluates the performance of binary classifiers for scalable link prediction with limited data. Our evaluation pipeline is made open source, and with this we aim to draw more attention of the community towards an important issue of transparency and reproducibility of the neural embeddings evaluations.