AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing

TL;DR

This paper introduces AFTER, a novel method for reducing object hallucinations in large vision-language models by adaptively guiding internal activations with factual textual semantics, significantly improving trustworthiness.

Contribution

The paper proposes a new adaptive editing framework, AFTER, that effectively incorporates factual guidance to mitigate hallucinations in LVLMs, addressing limitations of previous methods.

Findings

Achieves up to 16.3% reduction in hallucinations on AMBER benchmark.

Demonstrates effectiveness across three widely adopted LVLMs.

Validates the approach through extensive experiments on standard benchmarks.

Abstract

Large Vision-Language Models (LVLMs) have achieved substantial progress in cross-modal tasks. However, due to language bias, LVLMs are susceptible to object hallucination, which can be primarily divided into category, attribute, and relation hallucination, significantly impeding the trustworthy AI applications. Editing the internal activations of LVLMs has shown promising effectiveness in mitigating hallucinations with minimal cost. However, previous editing approaches neglect the effective guidance offered by factual textual semantics, thereby struggling to explicitly mitigate language bias. To address these issues, we propose Adaptive Factual-guided Visual-Textual Editing for hallucination mitigation (AFTER), which comprises Factual-Augmented Activation Steering (FAS) and Query-Adaptive Offset Optimization (QAO), to adaptively guides the original biased activations towards factual semantics. Specifically, FAS is proposed to provide factual and general guidance for activation editing, thereby explicitly modeling the precise visual-textual associations. Subsequently, QAO introduces a query-aware offset estimator to establish query-specific editing from the general steering vector, enhancing the diversity and granularity of editing. Extensive experiments on standard hallucination benchmarks across three widely adopted LVLMs validate the efficacy of the proposed AFTER, notably achieving up to a 16.3% reduction of hallucination over baseline on the AMBER benchmark. Our code and data will be released for reproducibility.