Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework

TL;DR

This study systematically re-implements and evaluates 21 knowledge graph embedding models across multiple datasets, providing insights into reproducibility, best practices, and factors influencing model performance.

Contribution

It offers a large-scale, unified benchmarking framework for knowledge graph embeddings, highlighting the importance of configuration choices beyond architecture.

Findings

Model performance depends heavily on architecture, training, and inverse relation modeling.

Many models can achieve competitive results with proper configuration.

Reproducibility issues are prevalent, but can be mitigated with standardized implementations.

Abstract

The heterogeneity in recently published knowledge graph embedding models' implementations, training, and evaluation has made fair and thorough comparisons difficult. In order to assess the reproducibility of previously published results, we re-implemented and evaluated 21 interaction models in the PyKEEN software package. Here, we outline which results could be reproduced with their reported hyper-parameters, which could only be reproduced with alternate hyper-parameters, and which could not be reproduced at all as well as provide insight as to why this might be the case. We then performed a large-scale benchmarking on four datasets with several thousands of experiments and 24,804 GPU hours of computation time. We present insights gained as to best practices, best configurations for each model, and where improvements could be made over previously published best configurations. Our results highlight that the combination of model architecture, training approach, loss function, and the explicit modeling of inverse relations is crucial for a model's performances, and not only determined by the model architecture. We provide evidence that several architectures can obtain results competitive to the state-of-the-art when configured carefully. We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at https://github.com/pykeen/pykeen and https://github.com/pykeen/benchmarking