Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context

TL;DR

This paper investigates why LVLMs are more prone to hallucinations in longer responses, finding that reliance on context, not length itself, increases hallucination risk, and proposes a framework to induce, detect, and suppress hallucinations effectively.

Contribution

The paper introduces a novel induce-detect-suppress framework that actively manages hallucinations in LVLMs by leveraging context, providing new insights into the mechanisms behind hallucination susceptibility.

Findings

Hallucination risk correlates with increased reliance on context in longer responses.

The proposed framework significantly reduces hallucinations across benchmarks.

Insights validate the hypothesis that context reliance, not length alone, causes hallucinations.

Abstract

Large Vision-Language Models (LVLMs) have made significant progress in recent years but are also prone to hallucination issues. They exhibit more hallucinations in longer, free-form responses, often attributed to accumulated uncertainties. In this paper, we ask: Does increased hallucination result solely from length-induced errors, or is there a deeper underlying mechanism? After a series of preliminary experiments and findings, we suggest that the risk of hallucinations is not caused by length itself but by the increased reliance on context for coherence and completeness in longer responses. Building on these insights, we propose a novel "induce-detect-suppress" framework that actively induces hallucinations through deliberately designed contexts, leverages induced instances for early detection of high-risk cases, and ultimately suppresses potential object-level hallucinations during actual decoding. Our approach achieves consistent, significant improvements across all benchmarks, demonstrating its efficacy. The strong detection and improved hallucination mitigation not only validate our framework but, more importantly, re-validate our hypothesis on context. Rather than solely pursuing performance gains, this study aims to provide new insights and serves as a first step toward a deeper exploration of hallucinations in LVLMs' longer responses.