Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steering

TL;DR

This paper introduces Contrastive Neuron Steering, a novel method that analyzes and mitigates hallucinations in LVLMs by manipulating interpretable neurons, leading to improved visual grounding and reduced hallucinations.

Contribution

It presents a representation-level approach using sparse autoencoders and contrastive analysis to identify and control image-specific neurons, enhancing robustness and interpretability in LVLMs.

Findings

CNS reduces hallucinations across benchmarks

Selective neuron modulation improves visual grounding

Method is compatible with existing decoding techniques

Abstract

LVLMs achieve remarkable multimodal understanding and generation but remain susceptible to hallucinations. Existing mitigation methods predominantly focus on output-level adjustments, leaving the internal mechanisms that give rise to these hallucinations largely unexplored. To gain a deeper understanding, we adopt a representation-level perspective by introducing sparse autoencoders (SAEs) to decompose dense visual embeddings into sparse, interpretable neurons. Through neuron-level analysis, we identify distinct neuron types, including always-on neurons and image-specific neurons. Our findings reveal that hallucinations often result from disruptions or spurious activations of image-specific neurons, while always-on neurons remain largely stable. Moreover, selectively enhancing or suppressing image-specific neurons enables controllable intervention in LVLM outputs, improving visual grounding and reducing hallucinations. Building on these insights, we propose Contrastive Neuron Steering (CNS), which identifies image-specific neurons via contrastive analysis between clean and noisy inputs. CNS selectively amplifies informative neurons while suppressing perturbation-induced activations, producing more robust and semantically grounded visual representations. This not only enhances visual understanding but also effectively mitigates hallucinations. By operating at the prefilling stage, CNS is fully compatible with existing decoding-stage methods. Extensive experiments on both hallucination-focused and general multimodal benchmarks demonstrate that CNS consistently reduces hallucinations while preserving overall multimodal understanding.