HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing

TL;DR

This paper introduces HIME, a training-free model editing method that selectively reduces object hallucinations in LVLMs by targeting sensitive layers, significantly improving factual accuracy without extra computational costs.

Contribution

The paper presents a systematic layer-wise analysis of LVLMs, introduces the HIS metric, and proposes HIME, a novel layer-adaptive editing approach to mitigate hallucinations effectively.

Findings

HIME reduces hallucinations by 61.8% on average.

Layer-wise differences influence hallucination susceptibility.

HIME preserves pre-trained knowledge without additional overhead.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal understanding capabilities, yet they remain prone to object hallucination, where models describe non-existent objects or attribute incorrect factual information, raising serious concerns for reliable real-world deployment. While fine-tuning is a commonly adopted mitigation strategy, its high computational cost and practical difficulty motivate the need for training-free alternatives, among which model editing has recently emerged as a promising direction. However, indiscriminate editing risks disrupting the rich implicit knowledge encoded in pre-trained LVLMs, leading to a fundamental question: how much intervention is necessary at each layer to suppress hallucinations while preserving pre-trained knowledge? To address this question, we present a systematic analysis of LVLM decoders built on three widely used large language model backbones-Qwen, LLaMA, and Vicuna-revealing clear layer-wise differences in susceptibility to object hallucination. Building on these insights, we introduce the Hallucination Insensitivity Score (HIS), a principled metric that quantifies each layer's sensitivity to hallucination and provides guidance for targeted intervention. Leveraging HIS, we propose Hallucination Insensitivity Model Editing (HIME), a simple yet effective layer-adaptive weight editing approach that selectively modifies latent features to suppress hallucinations while preserving pre-trained knowledge. Extensive experiments demonstrate that HIME reduces hallucinations by an average of 61.8% across open-ended generation benchmarks, including CHAIR, MME, and GPT-4V-aided evaluation, without introducing additional parameters, inference-time latency, or computational overhead.