Hallucination-aware intermediate representation edit in large vision-language models

TL;DR

This paper introduces a dynamic hallucination detection and editing framework for large vision-language models, significantly reducing hallucinations with minimal extra computation.

Contribution

It proposes a novel method for detecting and editing hallucination representations in vision-language models, improving robustness and controllability.

Findings

Achieves state-of-the-art hallucination mitigation performance.

Operates with minimal additional computational cost.

Demonstrates effective hallucination elimination and controllability.

Abstract

Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available at https://github.com/ASGO-MM/HIRE