IntelliGraphs: Datasets for Benchmarking Knowledge Graph Generation

TL;DR

IntelliGraphs introduces five new Knowledge Graph datasets with semantic subgraphs for benchmarking subgraph inference, highlighting the limitations of existing models in capturing underlying semantics and encouraging more semantically aware approaches.

Contribution

The paper presents IntelliGraphs, a new benchmark with datasets containing semantically rich subgraphs and evaluates baseline models, revealing their inability to capture semantics.

Findings

Baseline models fail to capture semantic information.

IntelliGraphs datasets enable evaluation of semantic understanding.

Benchmark encourages development of semantically aware models.

Abstract

Knowledge Graph Embedding (KGE) models are used to learn continuous representations of entities and relations. A key task in the literature is predicting missing links between entities. However, Knowledge Graphs are not just sets of links but also have semantics underlying their structure. Semantics is crucial in several downstream tasks, such as query answering or reasoning. We introduce the subgraph inference task, where a model has to generate likely and semantically valid subgraphs. We propose IntelliGraphs, a set of five new Knowledge Graph datasets. The IntelliGraphs datasets contain subgraphs with semantics expressed in logical rules for evaluating subgraph inference. We also present the dataset generator that produced the synthetic datasets. We designed four novel baseline models, which include three models based on traditional KGEs. We evaluate their expressiveness and show that these models cannot capture the semantics. We believe this benchmark will encourage the development of machine learning models that emphasize semantic understanding.