Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning

TL;DR

This paper introduces IECD$^2$, a dual-stream decoding framework that improves grounded vision-language reasoning by balancing language expressiveness with visual evidence fidelity, reducing hallucinations.

Contribution

The paper proposes a novel contrastive dual-stream decoding method that adaptively fuses instruction-driven and evidence-driven token probabilities for better grounded reasoning.

Findings

IECD$^2$ improves task accuracy across multiple datasets.

The method reduces hallucinations in generated outputs.

Demonstrates consistent performance gains over state-of-the-art approaches.

Abstract

Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning, yet they frequently generate fluent outputs that are weakly grounded in visual evidence. Prior works have shown that instruction prompting further worsens this issue by amplifying language priors, especially when the visual signal is uncertain or ambiguous. To address this challenge, we propose a decoding framework that explicitly balances linguistic informativeness and visual faithfulness during generation. Our method, Instruction-Evidence Contrastive Dual-Stream Decoding (IECD$^2$), maintains two parallel probability distribution of tokens at each decoding step: an instruction-driven stream that promotes expressive and informative responses, and an evidence-driven stream that enforces strict grounding in the image. These two streams are adaptively fused using a symmetric KL-based contrastive gate, which suppresses tokens favored by language priors but unsupported by visual evidence, while preserving them when both distributions agree. We evaluate IECD$^2$ on multiple datasets spanning various generative vision-language reasoning tasks such as captioning and visual question answering on multiple datasets such as, POPE, MME, VQAv2, AMBER, and MSCOCO. IECD$^2$ demonstrates consistent improvements in task accuracy and reasoning performance with substantial reduction in hallucination compared to state-of-the-art decoding approaches.