Document Reconstruction Unlocks Scalable Long-Context RLVR

TL;DR

This paper introduces an unsupervised document reconstruction method that improves the long-context understanding of large language models by training them to reconstruct missing document parts, reducing reliance on costly annotations.

Contribution

It proposes a novel unsupervised reinforcement learning approach for long-context modeling by reconstructing documents, enhancing LLMs' global coherence without heavy supervision.

Findings

Significant performance gains on RULER benchmark.

Reasonable improvements on LongBench v2 without manual QA data.

Extensive ablation studies validate the method's effectiveness.

Abstract

Reinforcement Learning with Verifiable Rewards~(RLVR) has become a prominent paradigm to enhance the capabilities (i.e.\ long-context) of Large Language Models~(LLMs). However, it often relies on gold-standard answers or explicit evaluation rubrics provided by powerful teacher models or human experts, which are costly and time-consuming. In this work, we investigate unsupervised approaches to enhance the long-context capabilities of LLMs, eliminating the need for heavy human annotations or teacher models' supervision. Specifically, we first replace a few paragraphs with special placeholders in a long document. LLMs are trained through reinforcement learning to reconstruct the document by correctly identifying and sequencing missing paragraphs from a set of candidate options. This training paradigm enables the model to capture global narrative coherence, significantly boosting long-context performance. We validate the effectiveness of our method on two widely used benchmarks, RULER and LongBench~v2. While acquiring noticeable gains on RULER, it can also achieve a reasonable improvement on LongBench~v2 without any manually curated long-context QA data. Furthermore, we conduct extensive ablation studies to analyze the impact of reward design, data curation strategies, training schemes, and data scaling effects on model performance. We publicly release our code, data, and models.