Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models

TL;DR

This paper introduces Mixture of Decoding (MoD), an adaptive decoding strategy for large vision-language models that reduces hallucinations by evaluating and adjusting based on attention correctness, leading to improved performance.

Contribution

The paper presents a novel adaptive decoding method, MoD, that dynamically adjusts decoding strategies based on attention correctness to mitigate hallucinations in LVLMs.

Findings

MoD significantly reduces hallucinations in LVLMs.

MoD outperforms existing decoding methods on multiple benchmarks.

The approach effectively distinguishes correct and incorrect attention during decoding.

Abstract

Large Vision-Language Models (LVLMs) have exhibited impressive capabilities across various visual tasks, yet they remain hindered by the persistent challenge of hallucinations. To address this critical issue, we propose Mixture of Decoding (MoD), a novel approach for hallucination mitigation that dynamically adapts decoding strategies by evaluating the correctness of the model's attention on image tokens. Specifically, MoD measures the consistency between outputs generated from the original image tokens and those derived from the model's attended image tokens, to distinguish the correctness aforementioned. If the outputs are consistent, indicating correct attention, MoD employs a complementary strategy to amplify critical information. Conversely, if the outputs are inconsistent, suggesting erroneous attention, MoD utilizes a contrastive strategy to suppress misleading information. Extensive experiments demonstrate that MoD significantly outperforms existing decoding methods across multiple mainstream benchmarks, effectively mitigating hallucinations in LVLMs. The code is available at https://github.com/xlchen0205/MoD.