Runtime Performances Benchmark for Knowledge Graph Embedding Methods

TL;DR

This paper evaluates the runtime performance of state-of-the-art knowledge graph embedding methods, focusing on memory and execution time, and compares architectures like CPU and GPU using a new benchmarking framework.

Contribution

It introduces a framework for benchmarking KGE implementations across different graph properties and architectures, highlighting optimization strategies and hardware effects.

Findings

GPU outperforms CPU for KGE tasks

Multithreading efficiency decreases with more threads on CPU

RAM usage depends on graph type, not model or architecture

Abstract

This paper wants to focus on providing a characterization of the runtime performances of state-of-the-art implementations of KGE alghoritms, in terms of memory footprint and execution time. Despite the rapidly growing interest in KGE methods, so far little attention has been devoted to their comparison and evaluation; in particular, previous work mainly focused on performance in terms of accuracy in specific tasks, such as link prediction. To this extent, a framework is proposed for evaluating available KGE implementations against graphs with different properties, with a particular focus on the effectiveness of the adopted optimization strategies. Graphs and models have been trained leveraging different architectures, in order to enlighten features and properties of both models and the architectures they have been trained on. Some results enlightened with experiments in this document are the fact that multithreading is efficient, but benefit deacreases as the number of threads grows in case of CPU. GPU proves to be the best architecture for the given task, even if CPU with some vectorized instructions still behaves well. Finally, RAM utilization for the loading of the graph never changes between different architectures and depends only on the type of graph, not on the model.