How Does Response Length Affect Long-Form Factuality

TL;DR

This paper investigates how the length of responses from large language models impacts their factual accuracy, revealing that longer responses tend to be less factual due to facts exhaustion.

Contribution

The study introduces a cost-effective, bi-level evaluation framework for long-form factuality and identifies facts exhaustion as the main reason for factual decline in longer responses.

Findings

Longer responses have lower factual precision.

Facts exhaustion is the primary cause of factual degradation.

Error propagation and long context are less influential.

Abstract

Large language models (LLMs) are widely used for long-form text generation. However, factual errors in the responses would undermine their reliability. Despite growing attention to LLM factuality, the effect of response length on factuality remains underexplored. In this work, we systematically investigate this relationship by first introducing an automatic and bi-level long-form factuality evaluation framework, which achieves high agreement with human annotations while being cost-effective. Using this framework, we conduct controlled experiments and find that longer responses exhibit lower factual precision, confirming the presence of length bias. To explain this phenomenon, we empirically examine three hypotheses: error propagation, long context, and facts exhaustion. Our results reveal that facts exhaustion, where the model gradually exhausts more reliable knowledge, is the primary cause of factual degradation, rather than the other two hypotheses.