Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction

TL;DR

This paper introduces MESA, a plug-and-play framework that reduces hallucinations in large vision-language models by selectively intervening in latent space, maintaining original generation behavior.

Contribution

MESA offers a novel, controlled latent intervention method that mitigates hallucinations without disrupting the intrinsic generation behavior of LVLMs.

Findings

MESA significantly reduces hallucinations across various benchmarks.

It preserves the original token distribution better than prior methods.

Outperforms existing approaches in multiple LVLM families.

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success across cross-modal tasks but remain hindered by hallucinations, producing textual outputs inconsistent with visual content. Existing methods mitigate hallucinations but often alter generation behavior, resulting in shorter outputs and shifted token distributions, especially in latent space steering approaches. We identify that this issue stems from entangled steering signals, where suppressing hallucinations inadvertently disrupts the model's intrinsic generation behavior. To address this, we propose MESA, an effective plug-and-play framework that performs controlled and selective latent intervention for hallucination mitigation. Specifically, MESA targets hallucination-relevant responses while preserving the model's original token distribution, enabling effective hallucination reduction without compromising generation behavior. Extensive experiments across diverse generative and discriminative benchmarks demonstrate that MESA consistently reduces hallucinations while better preserving generation behavior, outperforming prior methods across multiple LVLM families.