Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention

TL;DR

This paper introduces Vision-Guided Attention (VGA), a training-free method that improves the visual grounding of multimodal large language models, significantly reducing hallucinations during inference.

Contribution

VGA constructs precise visual grounding and guides model focus dynamically, achieving state-of-the-art dehallucination performance without additional training.

Findings

VGA reduces hallucinations across multiple benchmarks.

VGA is compatible with efficient attention methods like FlashAttention.

Each token in VGA requires only a single forward pass.

Abstract

Visual attention serves as the primary mechanism through which MLLMs interpret visual information; however, its limited localization capability often leads to hallucinations. We observe that although MLLMs can accurately extract visual semantics from visual tokens, they fail to fully leverage this advantage during subsequent inference. To address this limitation, we propose Vision-Guided Attention (VGA), a training-free method that first constructs precise visual grounding by exploiting the semantic content of visual tokens, and then uses this grounding to guide the model's focus toward relevant visual regions. In image captioning, VGA further refines this guidance dynamically during generation by suppressing regions that have already been described. In VGA, each token undergoes only a single forward pass, introducing a negligible latency overhead. In addition, VGA is fully compatible with efficient attention implementations such as FlashAttention. Extensive experiments across diverse MLLMs and multiple hallucination benchmarks demonstrate that VGA achieves state-of-the-art dehallucination performance. Further analysis confirms that explicit visual guidance plays a crucial role in enhancing the visual understanding capabilities of MLLMs.