The Reasoning Bottleneck in Graph-RAG: Structured Prompting and Context Compression for Multi-Hop QA

TL;DR

This paper identifies reasoning bottlenecks in Graph-RAG systems for multi-hop QA and proposes structured prompting and context compression techniques that significantly improve accuracy and efficiency, enabling smaller models to outperform larger baselines.

Contribution

Introduces SPARQL chain-of-thought prompting and graph-walk compression to enhance reasoning and reduce context size in Graph-RAG systems for multi-hop question answering.

Findings

SPARQL CoT improves accuracy by up to 14 percentage points.

Graph-walk compression reduces context size by ~60%.

Smaller models with augmentations outperform larger unaugmented models.

Abstract

Graph-RAG systems achieve strong multi-hop question answering by indexing documents into knowledge graphs, but strong retrieval does not guarantee strong answers. Evaluating KET-RAG, a leading Graph-RAG system, on three multi-hop QA benchmarks (HotpotQA, MuSiQue, 2WikiMultiHopQA), we find that 77% to 91% of questions have the gold answer in the retrieved context, yet accuracy is only 35% to 78%, and 73% to 84% of errors are reasoning failures. We propose two augmentations: (i) SPARQL chain-of-thought prompting, which decomposes questions into triple-pattern queries aligned with the entity-relationship context, and (ii) graph-walk compression, which compresses the context by ~60% via knowledge-graph traversal with no LLM calls. SPARQL CoT improves accuracy by +2 to +14 pp; graph-walk compression adds +6 pp on average when paired with structured prompting on smaller models. Surprisingly, we show that, with question-type routing, a fully augmented budget open-weight Llama-8B model matches or exceeds the unaugmented Llama-70B baseline on all three benchmarks at ~12x lower cost. A replication on LightRAG confirms that our augmentations transfer across Graph-RAG systems.