Is Complex Query Answering Really Complex?

TL;DR

This paper critically examines the perceived complexity of complex query answering on knowledge graphs, revealing that existing benchmarks are too simplistic and proposing more challenging tests that better reflect real-world reasoning tasks.

Contribution

The authors demonstrate that current benchmarks underestimate CQA complexity and introduce new benchmarks requiring multi-hop reasoning to better evaluate models.

Findings

Most queries in existing benchmarks are reducible to simpler problems.

State-of-the-art models perform poorly on the new, more challenging benchmarks.

Current CQA methods need significant improvement for real-world applicability.

Abstract

Complex query answering (CQA) on knowledge graphs (KGs) is gaining momentum as a challenging reasoning task. In this paper, we show that the current benchmarks for CQA might not be as complex as we think, as the way they are built distorts our perception of progress in this field. For example, we find that in these benchmarks, most queries (up to 98% for some query types) can be reduced to simpler problems, e.g., link prediction, where only one link needs to be predicted. The performance of state-of-the-art CQA models decreases significantly when such models are evaluated on queries that cannot be reduced to easier types. Thus, we propose a set of more challenging benchmarks composed of queries that require models to reason over multiple hops and better reflect the construction of real-world KGs. In a systematic empirical investigation, the new benchmarks show that current methods leave much to be desired from current CQA methods.