Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

TL;DR

This paper introduces an internal self-correction mechanism using layer attention in large vision-language models, significantly reducing hallucinations by refining generated content during inference without external signals.

Contribution

It proposes a novel internal self-correction method with layer attention that enhances visual grounding in LVLMs, requiring minimal additional parameters.

Findings

Improves hallucination mitigation across multiple benchmarks

Enhances visual grounding without external correction signals

Requires only 0.2M to 0.1M additional parameters

Abstract

Although Large Vision-Language Models (LVLMs) have made substantial progress, hallucination, where generated text is not grounded in the visual input, remains a challenge. As LVLMs become stronger, previously reported hallucination patterns, such as linguistic bias and overthinking phenomenon, become far less consistent, making the corresponding mitigation techniques substantially less effective. In this paper, we introduce an Internal self-Correction mechanism utilizing Layer Attention (ICLA) that operates directly on hidden states during generation. Each layer selectively retrieves information from all preceding layers through a diagonal cross-layer attention mechanism, enabling self-refinement without any external correction signals. With introducing and training only 0.2M and 0.1M additional parameters on LLaVA1.5-7B and Qwen2.5-VL-7B, \ours consistently improves visual grounding across multiple hallucination benchmarks, demonstrating its effectiveness for more advanced LVLMs.