GRAD: Graph-Retrieved Adaptive Decoding for Hallucination Mitigation

TL;DR

GRAD is a decoding-time method that improves large language model factuality by grounding generation in corpus-derived evidence using a graph-based approach, without retraining.

Contribution

Introduces Graph-Retrieved Adaptive Decoding (GRAD), a novel, lightweight decoding technique that leverages corpus-derived token transition graphs to mitigate hallucinations in LLMs.

Findings

Achieves up to 9.7% higher intrinsic accuracy.

Reduces hallucination rates by 8.6%.

Improves correctness by 6.9% over greedy decoding.

Abstract

Hallucination mitigation remains a persistent challenge for large language models (LLMs), even as model scales grow. Existing approaches often rely on external knowledge sources, such as structured databases or knowledge graphs, accessed through prompting or retrieval. However, prompt-based grounding is fragile and domain-sensitive, while symbolic knowledge integration incurs heavy retrieval and formatting costs. Motivated by knowledge graphs, we introduce Graph-Retrieved Adaptive Decoding (GRAD), a decoding-time method that grounds generation in corpus-derived evidence without retraining. GRAD constructs a sparse token transition graph by accumulating next-token logits across a small retrieved corpus in a single forward pass. During decoding, graph-retrieved logits are max-normalized and adaptively fused with model logits to favor high-evidence continuations while preserving fluency. Across three models and a range of question-answering benchmarks spanning intrinsic, extrinsic hallucination, and factuality tasks, GRAD consistently surpasses baselines, achieving up to 9.7$\%$ higher intrinsic accuracy, 8.6$\%$ lower hallucination rates, and 6.9$\%$ greater correctness compared to greedy decoding, while attaining the highest truth--informativeness product score among all methods. GRAD offers a lightweight, plug-and-play alternative to contrastive decoding and knowledge graph augmentation, demonstrating that statistical evidence from corpus-level token transitions can effectively steer generation toward more truthful and verifiable outputs.