Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation

TL;DR

This paper introduces LTS-FS, a layerwise feature steering framework guided by attribution scores to mitigate hallucinations in Large Vision-Language Models without degrading overall performance.

Contribution

It proposes a novel attribution-based, layerwise feature steering method that selectively targets hallucination-relevant layers in LVLMs.

Findings

LTS-FS significantly reduces hallucinations across multiple benchmarks.

The method preserves the performance of LVLMs on general tasks.

Attribution scores effectively identify hallucination-relevant layers.

Abstract

Despite the significant advancements in Large Vision-Language Models (LVLMs), their tendency to generate hallucinations undermines reliability and restricts broader practical deployment. Among the hallucination mitigation methods, feature steering emerges as a promising approach that reduces erroneous outputs in LVLMs without increasing inference costs. However, current methods apply uniform feature steering across all layers. This heuristic strategy ignores inter-layer differences, potentially disrupting layers unrelated to hallucinations and ultimately leading to performance degradation on general tasks. In this paper, we propose Locate-Then-Sparsify for Feature Steering (LTS-FS), a plug-and-play framework which controls the steering intensity according to the hallucination relevance of each layer. We first construct a dataset comprising token-level and sentence-level hallucination cases. Based on this dataset, we introduce an attribution method based on causal interventions to quantify the hallucination relevance of each layer. With the attribution scores across layers, we propose a layerwise strategy that converts these scores into feature steering intensities for individual layers, enabling more precise adjustments specifically on hallucination-relevant layers. Extensive experiments across multiple LVLMs and benchmarks demonstrate that LTS-FS effectively mitigates hallucination while preserving strong performance. Codes are available at https://github.com/huttersadan/LTS-FS.