SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions
Shicheng Liu, Sina J. Semnani, Harold Triedman, Jialiang Xu, Isaac Dan, Zhao, Monica S. Lam

TL;DR
This paper introduces SPINACH, a new challenging KBQA dataset from real-world queries and an in-context learning agent that achieves state-of-the-art results, demonstrating improved handling of complex, schema-exploratory questions.
Contribution
The paper presents a novel dataset from real-world discussions and an in-context learning KBQA agent that outperforms existing models on multiple benchmarks.
Findings
SPINACH dataset captures complex real-world KBQA queries.
The SPINACH agent achieves state-of-the-art results on multiple datasets.
Outperforms GPT-4-based KBQA systems by significant margins.
Abstract
Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas. We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsNatural Language Processing Techniques · Service-Oriented Architecture and Web Services · Topic Modeling
MethodsBalanced Selection · LLaMA
