Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models

TL;DR

This paper introduces Cross-Image Contrastive Decoding (CICD), a training-free method that reduces hallucinations in large vision-language models by selectively suppressing language priors using unrelated images, improving output accuracy.

Contribution

CICD leverages unrelated images for contrastive decoding and employs a dynamic selection mechanism to precisely suppress language priors without harming response quality.

Findings

Reduces hallucinations in LVLMs effectively

Improves image captioning accuracy

Generalizes across multiple benchmarks

Abstract

Over-reliance on language priors is a major cause of hallucinations in Large Vision-Language Models (LVLMs), often leading to outputs that are linguistically plausible but visually inconsistent. Recent studies have explored contrastive decoding as a training-free solution. However, these methods typically construct contrastive visual inputs by perturbing the original image, resulting in distorted contrastive distributions, incomplete contrastive signals, and excessive suppression of language priors. Motivated by the observation that language priors tend to remain consistent across different images, we propose Cross-Image Contrastive Decoding (CICD), a simple yet effective training-free method that uses unrelated images as contrastive visual inputs. To address the issue of over-suppressing language priors, which can negatively affect the quality of generated responses, we further introduce a dynamic selection mechanism based on the cross-image differences in model behavior. By selectively suppressing language priors, our method reduces hallucinations without compromising the model's performance. Extensive experiments across multiple benchmarks and LVLMs confirm the effectiveness and generalizability of CICD, particularly in image captioning, where language priors are especially dominant.