VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification

TL;DR

VASparse introduces an efficient token sparsification method during decoding to significantly reduce visual hallucinations in large vision-language models without extra training, maintaining speed and improving output faithfulness.

Contribution

The paper presents a novel visual-aware token sparsification strategy that effectively mitigates hallucinations in LVLMs during decoding, without additional training or post-processing.

Findings

VASparse reduces visual hallucinations across multiple benchmarks.

It maintains competitive decoding speed compared to existing methods.

VASparse achieves state-of-the-art hallucination mitigation performance.

Abstract

Large Vision-Language Models (LVLMs) may produce outputs that are unfaithful to reality, also known as visual hallucinations (VH), which significantly impedes their real-world usage. To alleviate VH, various decoding strategies have been proposed to enhance visual information. However, many of these methods may require secondary decoding and rollback, which significantly reduces inference speed. In this work, we propose an efficient plug-and-play decoding algorithm via Visual-Aware Sparsification (VASparse) from the perspective of token sparsity for mitigating VH. VASparse is inspired by empirical observations: (1) the sparse activation of attention in LVLMs, and (2) visual-agnostic tokens sparsification exacerbates VH. Based on these insights, we propose a novel token sparsification strategy that balances efficiency and trustworthiness. Specifically, VASparse implements a visual-aware token selection strategy during decoding to reduce redundant tokens while preserving visual context effectively. Additionally, we innovatively introduce a sparse-based visual contrastive decoding method to recalibrate the distribution of hallucinated outputs without the time overhead associated with secondary decoding. Subsequently, VASparse recalibrates attention scores to penalize attention sinking of LVLMs towards text tokens. Extensive experiments across four popular benchmarks confirm the effectiveness of VASparse in mitigating VH across different LVLM families without requiring additional training or post-processing. Impressively, VASparse achieves state-of-the-art performance for mitigating VH while maintaining competitive decoding speed. Code is available at https://github.com/mengchuang123/VASparse-github.