Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models

TL;DR

This paper introduces a phase-aware, training-free method to reduce hallucinations in vision-language models by selectively suppressing low-attention tokens during inference, improving accuracy with minimal latency.

Contribution

It uncovers a three-phase attention structure in vision encoders and leverages this insight to develop a lightweight, effective hallucination mitigation technique.

Findings

Significantly reduces hallucination metrics across multiple LVLMs.

Operates with negligible additional inference latency.

Maintains competitive caption quality while suppressing unreliable tokens.

Abstract

Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dynamics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion. Our analysis reveals that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time intervention that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a single forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucination metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency.