On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility

TL;DR

This paper introduces VOLTBench, a benchmark for measuring length volatility in long-form text generation by LLMs, analyzes internal causes, and proposes GLoBo, a decoding strategy that significantly stabilizes output length.

Contribution

It presents the first systematic benchmark for length volatility, identifies internal causes, and proposes a lightweight decoding method to improve stability without retraining.

Findings

VOLTBench reveals severe length volatility in mainstream models.

GLoBo reduces length volatility by 69%.

GLoBo increases mean output length by 148%.

Abstract

Large Language Models (LLMs) excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volatility of the output. This volatility not only leads to significant computational costs but also severely impacts the models' reliable application. To address this gap, our work unfolds in three stages: benchmarking, probing, and mitigation. We first propose the VOlatility in Long-form Text Benchmark (VOLTBench), a novel heterogeneous-task benchmark designed to systematically quantify the length volatility of long-form generation. Subsequently, by analyzing attention traces, we conduct an in-depth probe to identify several common internal patterns that cause this volatility. Finally, to mitigate long-form output volatility, we propose Stable Generation via Logits Boosting (GLoBo), a lightweight decoding-stage optimization strategy, designed to significantly enhance both the length accuracy and stability of long-form generation without additional training. Extensive experiments on VOLTBench provide the first systematic confirmation of severe long-form output instability in mainstream models and validate that our proposed method successfully improves the mean output length of the base model by 148% and reduces the length volatility by 69%, while maintaining high generation quality.