GRASP: Generic Reasoning And SPARQL Generation across Knowledge Graphs

TL;DR

GRASP introduces a zero-shot, large language model-based method for generating SPARQL queries from natural language, achieving state-of-the-art results across diverse knowledge graphs without fine-tuning.

Contribution

It presents a novel, fine-tuning-free approach that explores knowledge graphs through strategic SPARQL query execution guided by large language models.

Findings

Achieves state-of-the-art results on Wikidata benchmarks

Performs well on less common knowledge graphs

Effective across various language models and sizes

Abstract

We propose a new approach for generating SPARQL queries on RDF knowledge graphs from natural language questions or keyword queries, using a large language model. Our approach does not require fine-tuning. Instead, it uses the language model to explore the knowledge graph by strategically executing SPARQL queries and searching for relevant IRIs and literals. We evaluate our approach on a variety of benchmarks (for knowledge graphs of different kinds and sizes) and language models (of different scales and types, commercial as well as open-source) and compare it with existing approaches. On Wikidata we reach state-of-the-art results on multiple benchmarks, despite the zero-shot setting. On Freebase we come close to the best few-shot methods. On other, less commonly evaluated knowledge graphs and benchmarks our approach also performs well overall. We conduct several additional studies, like comparing different ways of searching the graphs, incorporating a feedback mechanism, or making use of few-shot examples.