SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions

TL;DR

This paper introduces SPINACH, a new challenging KBQA dataset from real-world queries and an in-context learning agent that achieves state-of-the-art results, demonstrating improved handling of complex, schema-exploratory questions.

Contribution

The paper presents a novel dataset from real-world discussions and an in-context learning KBQA agent that outperforms existing models on multiple benchmarks.

Findings

SPINACH dataset captures complex real-world KBQA queries.

The SPINACH agent achieves state-of-the-art results on multiple datasets.

Outperforms GPT-4-based KBQA systems by significant margins.

Abstract

Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas. We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in $F_1$, respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions. On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4-based KBQA agent, by at least 38.1% in $F_1$.