Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding

TL;DR

This paper introduces ATED, a training-free, adaptive token ensemble decoding method that reduces hallucinations in large vision-language models by dynamically aggregating multiple model predictions during inference.

Contribution

The paper presents ATED, a novel ensemble decoding framework that mitigates hallucinations without additional training, improving robustness of LVLMs in multimodal tasks.

Findings

ATES significantly reduces hallucinations compared to state-of-the-art methods.

The approach maintains fluency and relevance in generated descriptions.

Adaptive weighting improves model reliability at each decoding step.

Abstract

Large Vision-Language Models (LVLMs) have recently achieved impressive results in multimodal tasks such as image captioning and visual question answering. However, they remain prone to object hallucination -- generating descriptions of nonexistent or misidentified objects. Prior work has partially mitigated this via auxiliary training objectives or external modules, but challenges remain in terms of scalability, adaptability, and model independence. To address these limitations, we propose Adaptive Token Ensemble Decoding (ATED), a training-free, token-level ensemble framework that mitigates hallucination by aggregating predictions from multiple LVLMs during inference. ATED dynamically computes uncertainty-based weights for each model, reflecting their reliability at each decoding step. It also integrates diverse decoding paths to improve contextual grounding and semantic consistency. Experiments on standard hallucination detection benchmarks demonstrate that ATED significantly outperforms state-of-the-art methods, reducing hallucination without compromising fluency or relevance. Our findings highlight the benefits of adaptive ensembling and point to a promising direction for improving LVLM robustness in high-stakes applications. The code is available at https://github.com/jinlin2021/ATED.