SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation

TL;DR

SAGE is a decoding framework that reduces hallucinations in vision-language models by adaptively modulating self-attention based on sink tokens, improving grounding and content accuracy.

Contribution

It introduces a novel sink-aware decoding method that dynamically adjusts attention during generation to mitigate hallucinations without retraining or architecture changes.

Findings

Achieves an average of 10.65% improvement on MSCOCO

Achieves an average of 7.19% improvement on AMBER

Consistently outperforms existing decoding strategies in hallucination reduction

Abstract

Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.