MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction

TL;DR

This paper introduces MINT, a training-free decoding strategy that reduces hallucinations in large vision-language models by masking irrelevant image tokens and emphasizing key visual regions, leading to improved perception and reliability.

Contribution

MINT is a novel, training-free method that mitigates hallucinations in LVLMs by dynamically reducing attention to non-essential tokens and enhancing focus on key visual elements.

Findings

Achieves 4% reduction in hallucinations on benchmarks

Perceives 5% more visual points despite token reduction

Improves model reliability in high-stakes domains

Abstract

Hallucination has been a long-standing and inevitable problem that hinders the application of Large Vision-Language Models (LVLMs) in domains that require high reliability. Various methods focus on improvement depending on data annotations or training strategies, yet place less emphasis on LLM's inherent problems. To fill this gap, we delve into the attention mechanism of the decoding process in the LVLM. Intriguingly, our investigation uncovers the prevalent attention redundancy within the hierarchical architecture of the LVLM, manifesting as overextended image processing in deep layers and an overabundance of non-essential image tokens. Stemming from the observation, we thus propose MINT, a novel training-free decoding strategy, MItigating hallucinations via tokeN reducTion. Specifically, we dynamically intensify the LVLM's local perception capability by masking its attention to irrelevant image tokens. In addition, we use contrastive decoding that pushes the model to focus more on those key image regions. Our full method aims to guide the model in concentrating more on key visual elements during generation. Extensive experimental results on several popular public benchmarks show that our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models. Meanwhile, our approach is demonstrated to make the model perceive 5% more visual points even though we reduce a suite of image tokens.