ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models

TL;DR

ONLY is a training-free, one-layer intervention method that significantly reduces hallucinations in large vision-language models, enabling efficient real-time responses without multiple queries.

Contribution

The paper introduces ONLY, a novel single-query, one-layer intervention technique that effectively mitigates hallucinations in LVLMs without additional training or multiple queries.

Findings

OUTPERFORMS state-of-the-art methods across benchmarks

Requires only a single query and minimal computational resources

Enhances textual output by amplifying crucial information

Abstract

Recent Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses. Although they have achieved remarkable performance across a range of multi-modal tasks, they face the persistent challenge of hallucination, which introduces practical weaknesses and raises concerns about their reliable deployment in real-world applications. Existing work has explored contrastive decoding approaches to mitigate this issue, where the output of the original LVLM is compared and contrasted with that of a perturbed version. However, these methods require two or more queries that slow down LVLM response generation, making them less suitable for real-time applications. To overcome this limitation, we propose ONLY, a training-free decoding approach that requires only a single query and a one-layer intervention during decoding, enabling efficient real-time deployment. Specifically, we enhance textual outputs by selectively amplifying crucial textual information using a text-to-visual entropy ratio for each token. Extensive experimental results demonstrate that our proposed ONLY consistently outperforms state-of-the-art methods across various benchmarks while requiring minimal implementation effort and computational cost. Code is available at https://github.com/zifuwan/ONLY.