Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors

TL;DR

This paper identifies that hallucinations in multimodal large language models originate from deep layer attention regressions and proposes CLVA, a training-free method that uses cross-layer visual anchors to improve visual attention accuracy and reduce hallucinations.

Contribution

The paper introduces CLVA, a novel training-free approach that leverages cross-layer visual anchors to mitigate hallucinations by stabilizing deep layer attention in multimodal models.

Findings

CLVA effectively reduces hallucinations across various architectures.

The method improves attention stability without increasing computational costs.

Experimental results show enhanced model reliability and interpretability.

Abstract

Multimodal Large Language Models often suffer from object hallucination. While existing research utilizes attention enhancement and visual retracing, we find these works lack sufficient interpretability regarding attention drift in final model stages. In this paper, we investigate the layer wise evolution of visual features and discover that hallucination stems from deep layer attention regressing toward initial visual noise from early layers. We observe that output reliability depends on acquiring visual anchors at intermediate layers rather than final layers. Based on these insights, we propose CLVA, which stands for Cross-Layer Visual Anchors, a training free method that reinforces critical mid layer features while suppressing regressive noise. This approach effectively pulls deep layer attention back to correct visual regions by utilizing essential anchors captured from attention dynamics. We evaluate our method across diverse architectures and benchmarks, demonstrating outstanding performance without significant increase in computational time and GPU memory.