Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models

TL;DR

This paper introduces AvisC, a test-time method that reduces hallucinations in large vision-language models by recalibrating attention to irrelevant image tokens without altering the model's architecture.

Contribution

Proposes a novel test-time calibration method, AvisC, that dynamically adjusts attention to mitigate hallucinations in LVLMs without changing their core structure.

Findings

Significantly reduces hallucinations on benchmark datasets.

Improves alignment between attention focus and relevant image regions.

Enhances the reliability of LVLMs in visual understanding tasks.

Abstract

Large Vision Language Models (LVLMs) demonstrate strong capabilities in visual understanding and description, yet often suffer from hallucinations, attributing incorrect or misleading features to images. We observe that LVLMs disproportionately focus on a small subset of image tokens--termed blind tokens--which are typically irrelevant to the query (e.g., background or non-object regions). We hypothesize that such attention misalignment plays a key role in generating hallucinated responses. To mitigate this issue, we propose Attentional Vision Calibration (AvisC), a test-time approach that dynamically recalibrates the influence of blind tokens without modifying the underlying attention mechanism. AvisC first identifies blind tokens by analyzing layer-wise attention distributions over image tokens, then employs a contrastive decoding strategy to balance the influence of original and blind-token-biased logits. Experiments on standard benchmarks, including POPE, MME, and AMBER, demonstrate that AvisC effectively reduces hallucinations in LVLMs.