Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization

TL;DR

This paper studies the tendency of hallucinations to occur more frequently in the latter parts of long responses generated by LLMs, especially in long document summarization, and explores mitigation strategies.

Contribution

It identifies the positional bias of hallucinations in long text generation and investigates underlying causes and mitigation methods for improved faithfulness.

Findings

Hallucinations are concentrated in the latter parts of long responses.

Attention and decoding dynamics contribute to positional hallucination bias.

Mitigation strategies can reduce hallucinations in the concluding segments.

Abstract

Large Language Models (LLMs) have significantly advanced text generation capabilities, including tasks like summarization, often producing coherent and fluent outputs. However, faithfulness to source material remains a significant challenge due to the generation of hallucinations. While extensive research focuses on detecting and reducing these inaccuracies, less attention has been paid to the positional distribution of hallucination within generated text, particularly in long outputs. In this work, we investigate where hallucinations occur in LLM-based long response generation, using long document summarization as a key case study. Focusing on the challenging setting of long context-aware long response generation, we find a consistent and concerning phenomenon: hallucinations tend to concentrate disproportionately in the latter parts of the generated long response. To understand this bias, we explore potential contributing factors related to the dynamics of attention and decoding over long sequences. Furthermore, we investigate methods to mitigate this positional hallucination, aiming to improve faithfulness specifically in the concluding segments of long outputs.