Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

TL;DR

This paper introduces Ensemble Decoding, a novel method that reduces object hallucination in vision-language models by splitting images, combining logits with attention-guided weights, and calibrating plausibility, achieving state-of-the-art results.

Contribution

The paper presents a new ensemble decoding strategy with adaptive plausibility constraints and a fast variant, improving hallucination mitigation in vision-language models.

Findings

Achieves state-of-the-art performance on hallucination benchmarks.

Effectively reduces object hallucination in image captioning and VQA.

Provides a scalable, training-free approach with speed-optimized variant.

Abstract

Recent advancements in Large Vision-Language Models (LVLMs) have significantly expanded their utility in tasks like image captioning and visual question answering. However, they still struggle with object hallucination, where models generate descriptions that inaccurately reflect the visual content by including nonexistent objects or misrepresenting existing ones. While previous methods, such as data augmentation and training-free approaches, strive to tackle this issue, they still encounter scalability challenges and often depend on additional external modules. In this work, we propose Ensemble Decoding (ED), a novel strategy that splits the input image into sub-images and combines logit distributions by assigning weights through the attention map. Furthermore, we introduce ED adaptive plausibility constraint to calibrate logit distribution and FastED, a variant designed for speed-critical applications. Extensive experiments across hallucination benchmarks demonstrate that our proposed method achieves state-of-the-art performance, validating the effectiveness of our approach.