CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models

TL;DR

CRoPS is a training-free framework that mitigates hallucinations in vision-language models by using selective token removal and generalized contrastive decoding, significantly improving reliability across multiple benchmarks.

Contribution

It introduces a novel hallucination modeling approach with selective token removal and a generalized contrastive decoding method, outperforming existing training-free techniques.

Findings

Improves CHAIR scores by 20%.

Achieves consistent gains across six benchmarks.

Outperforms state-of-the-art training-free methods.

Abstract

Despite the rapid success of Large Vision-Language Models (LVLMs), a persistent challenge is their tendency to generate hallucinated content, undermining reliability in real-world use. Existing training-free methods address hallucinations but face two limitations: (i) they rely on narrow assumptions about hallucination sources, and (ii) their effectiveness declines toward the end of generation, where hallucinations are most likely to occur. A common strategy is to build hallucinated models by completely or partially removing visual tokens and contrasting them with the original model. Yet, this alone proves insufficient, since visual information still propagates into generated text. Building on this insight, we propose a novel hallucinated model that captures hallucination effects by selectively removing key text tokens. We further introduce Generalized Contrastive Decoding, which integrates multiple hallucinated models to represent diverse hallucination sources. Together, these ideas form CRoPS, a training-free hallucination mitigation framework that improves CHAIR scores by 20% and achieves consistent gains across six benchmarks and three LVLM families, outperforming state-of-the-art training-free methods.