IRT2: Inductive Linking and Ranking in Knowledge Graphs of Varying Scale

TL;DR

This paper introduces IRT2, an open benchmark for inductive link prediction in knowledge graphs of varying sizes, incorporating text mentions and ranking, to better reflect industrial challenges and evaluate neural models.

Contribution

The paper presents IRT2, a new benchmark dataset that includes small graphs, low-quality text mentions, and ranking tasks, addressing gaps in existing benchmarks for industrial knowledge graph applications.

Findings

Neural models outperform baselines as graph data decreases.

Neural approaches significantly improve ranking over sparse retrievers.

Performance gains are observed in low-data scenarios.

Abstract

We address the challenge of building domain-specific knowledge models for industrial use cases, where labelled data and taxonomic information is initially scarce. Our focus is on inductive link prediction models as a basis for practical tools that support knowledge engineers with exploring text collections and discovering and linking new (so-called open-world) entities to the knowledge graph. We argue that - though neural approaches to text mining have yielded impressive results in the past years - current benchmarks do not reflect the typical challenges encountered in the industrial wild properly. Therefore, our first contribution is an open benchmark coined IRT2 (inductive reasoning with text) that (1) covers knowledge graphs of varying sizes (including very small ones), (2) comes with incidental, low-quality text mentions, and (3) includes not only triple completion but also ranking, which is relevant for supporting experts with discovery tasks. We investigate two neural models for inductive link prediction, one based on end-to-end learning and one that learns from the knowledge graph and text data in separate steps. These models compete with a strong bag-of-words baseline. The results show a significant advance in performance for the neural approaches as soon as the available graph data decreases for linking. For ranking, the results are promising, and the neural approaches outperform the sparse retriever by a wide margin.