MACD: Model-Aware Contrastive Decoding via Counterfactual Data

TL;DR

MACD is a novel decoding strategy for Video-LLMs that reduces hallucinations by using model-guided counterfactual data to improve evidence-grounded content generation, especially in challenging visual scenarios.

Contribution

It introduces a model-aware counterfactual data construction method integrated with contrastive decoding to mitigate hallucinations in Video-LLMs.

Findings

Significantly reduces hallucinations across multiple benchmarks.

Maintains or improves task accuracy in diverse Video-LLMs.

Effective in scenarios with small, occluded, or co-occurring objects.

Abstract

Video language models (Video-LLMs) are prone to hallucinations, often generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing decoding methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for mitigating hallucination patterns. However, such a way is hard to control the visual cues that drive hallucination or well align with model weaknesses. We propose Model-aware Counterfactual Data based Contrastive Decoding (MACD), a new inference strategy that combines model-guided counterfactual construction with decoding. Our approach uses the Video-LLM's own feedback to identify object regions most responsible for hallucination, generating targeted counterfactual inputs at the object level rather than arbitrary frame or temporal modifications. These model-aware counterfactual data is then integrated into CD to enforce evidence-grounded token selection during decoding. Experiments on EventHallusion, MVBench, Perception-test and Video-MME show that MACD consistently reduces hallucination while maintaining or improving task accuracy across diverse Video-LLMs, including Qwen and InternVL families. The method is especially effective in challenging scenarios involving small, occluded, or co-occurring objects. Our code and data will be publicly released.