Numerical Literals in Link Prediction: A Critical Examination of Models and Datasets

TL;DR

This paper critically examines the effectiveness of models using numerical literals in link prediction on knowledge graphs, revealing underutilization of literal information and the need for better evaluation datasets.

Contribution

It introduces a new synthetic dataset and ablation strategies to evaluate how well models utilize numerical literals in link prediction tasks.

Findings

Many models underuse literal information

Models may rely on extra parameters rather than literals

Existing datasets may not effectively test literal utilization

Abstract

Link Prediction(LP) is an essential task over Knowledge Graphs(KGs), traditionally focussed on using and predicting the relations between entities. Textual entity descriptions have already been shown to be valuable, but models that incorporate numerical literals have shown minor improvements on existing benchmark datasets. It is unclear whether a model is actually better in using numerical literals, or better capable of utilizing the graph structure. This raises doubts about the effectiveness of these methods and about the suitability of the existing benchmark datasets. We propose a methodology to evaluate LP models that incorporate numerical literals. We propose i) a new synthetic dataset to better understand how well these models use numerical literals and ii) dataset ablations strategies to investigate potential difficulties with the existing datasets. We identify a prevalent trend: many models underutilize literal information and potentially rely on additional parameters for performance gains. Our investigation highlights the need for more extensive evaluations when releasing new models and datasets.