Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs

TL;DR

This paper introduces Owl, a causally-grounded framework for reducing object hallucinations in LVLMs by modeling attention interactions, quantifying modality contributions, and dynamically intervening during decoding, leading to state-of-the-art results.

Contribution

The paper proposes a novel causally-grounded attention intervention framework with a new metric VTACR and a dual-path decoding strategy to effectively mitigate hallucinations in LVLMs.

Findings

Owl significantly reduces hallucinations on POPE and CHAIR benchmarks.

VTACR correlates with hallucination likelihood, guiding interventions.

Dual-path contrastive decoding improves faithfulness without sacrificing understanding.

Abstract

Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL