AdaVBoost: Mitigating Hallucinations in LVLMs via Token-Level Adaptive Visual Attention Boosting

TL;DR

AdaVBoost is a novel adaptive attention boosting method for LVLMs that dynamically adjusts visual attention at the token level to effectively reduce hallucinations during generation.

Contribution

It introduces Visual Grounding Entropy (VGE) to estimate hallucination risk and applies token-level adaptive attention boosting, improving hallucination mitigation over existing fixed-scaling methods.

Findings

Significantly reduces hallucinations across multiple LVLMs.

Outperforms baseline methods on various hallucination benchmarks.

Demonstrates effective token-level adaptive intervention during generation.

Abstract

Visual attention boosting has emerged as a promising direction for mitigating hallucinations in Large Vision-Language Models (LVLMs), where existing methods primarily focus on where to boost by applying a predefined scaling to the attention of method-specific visual tokens during autoregressive generation. In this paper, we identify a fundamental trade-off in these methods: a predefined scaling factor can be too weak at some generation steps, leaving hallucinations unresolved, yet too strong at others, leading to new hallucinations. Motivated by this finding, we propose AdaVBoost, a token-level visual attention boosting framework that adaptively determines how much attention to boost at each generation step. Specifically, we introduce Visual Grounding Entropy (VGE) to estimate hallucination risk, which leverages visual grounding as a complementary signal to capture evidence mismatches beyond entropy. Guided by VGE, AdaVBoost applies stronger visual attention boosting to high-risk tokens and weaker boosting to low-risk tokens, enabling token-level adaptive intervention at each generation step. Extensive experiments show that AdaVBoost significantly outperforms baseline methods across multiple LVLMs and hallucination benchmarks.