Why Retrieval-Augmented Generation Fails: A Graph Perspective

TL;DR

This paper investigates why Retrieval-Augmented Generation (RAG) systems often fail by analyzing internal information flow using circuit tracing and attribution graphs, leading to improved error detection and intervention methods.

Contribution

It introduces a novel graph-based analysis of RAG models, revealing structural differences between correct and incorrect predictions, and proposes targeted interventions to enhance evidence integration.

Findings

Correct predictions have deeper, more distributed evidence flow.

Failed predictions show shallower, fragmented evidence flow.

Graph-based interventions improve grounding and reduce errors.

Abstract

Retrieval-Augmented Generation (RAG) has become a powerful and widely used approach for improving large language models by grounding generation in retrieved evidence. However, RAG systems still produce incorrect answers in many cases. Why RAG fails despite having access to external information remains poorly understood. We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation. Using circuit tracing, we construct attribution graphs that model the flow of information through transformer layers during decoding. These graphs represent interactions among retrieved context, intermediate model activations, and generated tokens, providing a graph, circuit-level view of how external evidence is integrated into the model's reasoning process across multiple question answering benchmarks, we observe consistent structural differences: correct predictions exhibit deeper reasoning paths, more distributed evidence flow, and a more structured pattern of local connectivity, while failed predictions show shallower, fragmented, and overly concentrated evidence flow. Building on these findings, we develop a graph-based error detection framework that uses attribution-graph topology features. Furthermore, we show that attribution graphs enable targeted interventions. By reinforcing question-constrained evidence grounding, we reshape internal routing so that answer generation remains guided by the question, leading to more effective integration of retrieved information and fewer errors.