Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking

TL;DR

This paper identifies critical quality issues in popular KGQA datasets and introduces KGQAGen, an LLM-based framework that generates a more reliable and challenging benchmark to better evaluate KGQA systems.

Contribution

The paper presents KGQAGen, a novel LLM-in-the-loop framework for improving dataset quality and constructing a large-scale, reliable KGQA benchmark grounded in Wikidata.

Findings

Average factual correctness of existing datasets is only 57%.

State-of-the-art models perform poorly on the new KGQAGen-10k benchmark.

KGQAGen exposes limitations of current KGQA systems.

Abstract

Knowledge Graph Question Answering (KGQA) systems rely on high-quality benchmarks to evaluate complex multi-hop reasoning. However, despite their widespread use, popular datasets such as WebQSP and CWQ suffer from critical quality issues, including inaccurate or incomplete ground-truth annotations, poorly constructed questions that are ambiguous, trivial, or unanswerable, and outdated or inconsistent knowledge. Through a manual audit of 16 popular KGQA datasets, including WebQSP and CWQ, we find that the average factual correctness rate is only 57 %. To address these issues, we introduce KGQAGen, an LLM-in-the-loop framework that systematically resolves these pitfalls. KGQAGen combines structured knowledge grounding, LLM-guided generation, and symbolic verification to produce challenging and verifiable QA instances. Using KGQAGen, we construct KGQAGen-10k, a ten-thousand scale benchmark grounded in Wikidata, and evaluate a diverse set of KG-RAG models. Experimental results demonstrate that even state-of-the-art systems struggle on this benchmark, highlighting its ability to expose limitations of existing models. Our findings advocate for more rigorous benchmark construction and position KGQAGen as a scalable framework for advancing KGQA evaluation.