Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

TL;DR

This paper identifies attention imbalance as a key cause of object hallucination in LVLMs and introduces a rectification method that significantly reduces hallucinations and enhances model performance.

Contribution

It proposes the concept of attention imbalance, visualizes its patterns, and introduces AIR, a lightweight decoding-time method to rectify attention and mitigate hallucinations in LVLMs.

Findings

AIR reduces object hallucination rates by up to 35.1%.

It improves LVLMs' general capabilities by up to 15.9%.

Attention imbalance correlates strongly with hallucination occurrence.

Abstract

Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis. Through systematic empirical investigation, we identify that the imbalanced attention allocation, both across modalities (i.e., vision and language) and within modalities (among individual tokens), exhibits a strong causal correlation with the occurrence of object hallucination. Leveraging this insight, we introduce a novel concept termed attention imbalance, which not only quantifies the degree of attention disparity but also visually delineates the underlying patterns (e.g., over-attentiveness to irrelevant language tokens or under-attentiveness to discriminative visual features) that drive object hallucination. To mitigate object hallucination, we further propose Attention Imbalance Rectification (AIR), a lightweight decoding-time intervention method that reallocates attention weights and adjusts attention distributions to rectify modality-wise and token-wise imbalances. Extensive evaluations on four mainstream LVLMs and three benchmarks (CHAIR, POPE, and MM-Vet) with seven baselines demonstrate that AIR consistently reduces object hallucination rates, achieving up to a 35.1% reduction compared to the baselines, while improving up to 15.9% of LVLMs' general capability across diverse vision-language tasks.