Detecting Hallucinations in Graph Retrieval-Augmented Generation via Attention Patterns and Semantic Alignment

TL;DR

This paper introduces interpretability metrics and a hallucination detection method for GraphRAG systems, revealing how structural reliance and semantic misalignment in LLMs lead to hallucinations, and improving detection accuracy.

Contribution

It proposes two new interpretability metrics, PRD and SAS, and a post-hoc hallucination detector GGA, advancing understanding and mitigation of hallucinations in GraphRAG.

Findings

High PRD correlates with over-reliance on salient paths.

Low SAS indicates weak semantic grounding.

GGA outperforms baseline detectors in AUC and F1.

Abstract

Graph-based Retrieval-Augmented Generation (GraphRAG) enhances Large Language Models (LLMs) by incorporating external knowledge from linearized subgraphs retrieved from knowledge graphs. However, LLMs struggle to interpret the relational and topological information in these inputs, resulting in hallucinations that are inconsistent with the retrieved knowledge. To analyze how LLMs attend to and retain structured knowledge during generation, we propose two lightweight interpretability metrics: Path Reliance Degree (PRD), which measures over-reliance on shortest-path triples, and Semantic Alignment Score (SAS), which assesses how well the model's internal representations align with the retrieved knowledge. Through empirical analysis on a knowledge-based QA task, we identify failure patterns associated with over-reliance on salient paths and weak semantic grounding, as indicated by high PRD and low SAS scores. We further develop a lightweight post-hoc hallucination detector, Graph Grounding and Alignment (GGA), which outperforms strong semantic and confidence-based baselines across AUC and F1. By grounding hallucination analysis in mechanistic interpretability, our work offers insights into how structural limitations in LLMs contribute to hallucinations, informing the design of more reliable GraphRAG systems in the future.