Automating SPARQL Query Translations between DBpedia and Wikidata

TL;DR

This study evaluates the ability of large language models to automatically translate SPARQL queries between different knowledge graph schemas, revealing significant variation in performance and better results for Wikidata to DBpedia translations.

Contribution

It introduces a comprehensive evaluation of LLMs for SPARQL translation across multiple KGs, addressing a gap in KG interoperability research.

Findings

Wikidata to DBpedia translations perform better than DBpedia to Wikidata.

Model performance varies significantly across different LLMs and prompting strategies.

The study provides benchmarks for future research in KG schema translation.

Abstract

This paper investigates whether state-of-the-art Large Language Models (LLMs) can automatically translate SPARQL between popular Knowledge Graph (KG) schemas. We focus on translations between the DBpedia and Wikidata KG, and later on DBLP and OpenAlex KG. This study addresses a notable gap in KG interoperability research by rigorously evaluating LLM performance on SPARQL-to-SPARQL translation. Two benchmarks are assembled, where the first align 100 DBpedia-Wikidata queries from QALD-9-Plus; the second contains 100 DBLP queries aligned to OpenAlex, testing generalizability beyond encyclopaedic KGs. Three open LLMs: Llama-3-8B, DeepSeek-R1-Distill-Llama-70B, and Mistral-Large-Instruct-2407 are selected based on their sizes and architectures and tested with zero-shot, few-shot, and two chain-of-thought variants. Outputs were compared with gold answers, and resulting errors were categorized. We find that the performance varies markedly across models and prompting strategies, and that translations for Wikidata to DBpedia work far better than translations for DBpedia to Wikidata.