VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing

TL;DR

VCE is a post-hoc, label-free method that reduces object hallucinations in LVLMs by analyzing and editing model responses to visual perturbations, improving accuracy without fine-tuning.

Contribution

Introduces VCE, a novel visual contrastive editing technique that mitigates hallucinations in LVLMs without requiring labeled data or fine-tuning.

Findings

VCE significantly reduces object hallucination across benchmarks.

VCE maintains the original computational efficiency of LVLMs.

VCE operates as a scalable, label-free intervention.

Abstract

Large vision-language models (LVLMs) frequently suffer from Object Hallucination (OH), wherein they generate descriptions containing objects that are not actually present in the input image. This phenomenon is particularly problematic in real-world applications such as medical imaging and autonomous driving, where accuracy is critical. Recent studies suggest that the hallucination problem may stem from language priors: biases learned during pretraining that cause LVLMs to generate words based on their statistical co-occurrence. To mitigate this problem, we propose Visual Contrastive Editing (VCE), a novel post-hoc method that identifies and suppresses hallucinatory tendencies by analyzing the model's response to contrastive visual perturbations. Using Singular Value Decomposition (SVD), we decompose the model's activation patterns to isolate hallucination subspaces and apply targeted parameter edits to attenuate its influence. Unlike existing approaches that require fine-tuning or labeled data, VCE operates as a label-free intervention, making it both scalable and practical for deployment in resource-constrained settings. Experimental results demonstrate that VCE effectively reduces object hallucination across multiple benchmarks while maintaining the model's original computational efficiency.