Causal Decoding for Hallucination-Resistant Multimodal Large Language Models

TL;DR

This paper introduces a causal decoding method for multimodal large language models that effectively reduces object hallucination during vision-language tasks, improving reliability without sacrificing output quality.

Contribution

It presents a novel causal decoding framework that directly intervenes in generation dynamics to mitigate object hallucination in multimodal models.

Findings

Significantly lowers object hallucination rates in captioning and QA tasks.

Achieves state-of-the-art faithfulness in multimodal language models.

Maintains descriptive quality while reducing spurious object mentions.

Abstract

Multimodal Large Language Models (MLLMs) deliver detailed responses on vision-language tasks, yet remain susceptible to object hallucination (introducing objects not present in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation to curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencies, our approach reduces false object tokens while maintaining descriptive quality. Across captioning and QA benchmarks, our framework substantially lowers object-hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.