A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering

TL;DR

This paper presents an interactive human-in-the-loop system where LLMs generate and refine Cypher queries for knowledge graphs, enhancing explainability, accuracy, and accessibility in complex data domains.

Contribution

Introduces a novel framework combining LLMs and user interaction for generating and refining graph queries in natural language, improving knowledge graph querying.

Findings

Achieved high-quality query explanations and fault detection on a synthetic movie KG.

Demonstrated improved query generation on real-world KGs like Hyena and MaRDI.

Provided insights into model performance across different domains.

Abstract

Large Language Models (LLMs) excel at language understanding but remain limited in knowledge-intensive domains due to hallucinations, outdated information, and limited explainability. Text-based retrieval-augmented generation (RAG) helps ground model outputs in external sources but struggles with multi-hop reasoning. Knowledge Graphs (KGs), in contrast, support precise, explainable querying, yet require a knowledge of query languages. This work introduces an interactive framework in which LLMs generate and explain Cypher graph queries and users iteratively refine them through natural language. Applied to real-world KGs, the framework improves accessibility to complex datasets while preserving factual accuracy and semantic rigor and provides insight into how model performance varies across domains. Our core quantitative evaluation is a 90-query benchmark on a synthetic movie KG that measures query explanation quality and fault detection across multiple LLMs, complemented by two smaller real-life query-generation experiments on a Hyena KG and the MaRDI (Mathematical Research Data Initiative) KG.