Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs

TL;DR

This paper identifies a structural flaw in multimodal diffusion models that leads to hallucinations and introduces VISAGE, a training-free decoding method that improves visual grounding and reduces hallucinations by calibrating token ranking at inference.

Contribution

The paper reveals the cause of hallucinations in MDLLMs as an objective mismatch and proposes VISAGE, a novel inference-time calibration framework that enhances visual grounding without additional training.

Findings

VISAGE improves hallucination robustness by 8.59% on MMMU-val.

VISAGE enhances performance by 7.75% on HallusionBench.

The method maintains a bounded objective loss under estimation error.

Abstract

Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.