V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention

TL;DR

This paper introduces V-ITI, a novel framework that detects visual neglect in multimodal large language models and intervenes at inference time to reduce hallucinations, improving reliability without sacrificing performance.

Contribution

V-ITI is the first approach to detect visual neglect via head-level activation patterns and selectively intervene, effectively mitigating hallucinations in MLLMs during inference.

Findings

V-ITI reduces hallucinations across eight benchmarks.

The framework maintains task performance while decreasing hallucinations.

It is applicable to various MLLM architectures.

Abstract

Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations, producing content inconsistent with input visuals, that undermine reliability in precision-sensitive domains. This issue stems from a fundamental problem of visual neglect, where models fail to adequately prioritize input images. Existing methods typically alleviate hallucinations by intervening in the attention score or output logits, focusing on "how to intervene" but overlooking the prerequisite "when to intervene", which leads to the "over-intervention" problem and subsequently introduces new hallucinations and unnecessary computational overhead. To address this gap, we first investigate the mechanism of visual neglect and reveal it can be accurately detected via head-level activation patterns in MLLMs. We thus propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector that identifies visual neglect via head-level discriminative probes and a Visual Recall Intervenor that modulates activations with prestored visual activation information only when the visual neglect is detected. Extensive experiments across eight benchmarks and different MLLM families demonstrate that V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.