PolicyLong: Towards On-Policy Context Extension

TL;DR

PolicyLong introduces an on-policy data construction method for extending LLM context windows, dynamically adapting to the model's evolving capabilities to improve long-context learning.

Contribution

It proposes a novel iterative on-policy data screening approach that aligns training data with the model's current state, enhancing long-context performance.

Findings

PolicyLong outperforms EntropyLong and NExtLong across multiple benchmarks.

Significant gains observed at longer context lengths, e.g., +2.54 at 128K on RULER.

Dynamic on-policy data evolution improves model's ability to handle long contexts.

Abstract

Extending LLM context windows is hindered by scarce high-quality long-context data. Recent methods synthesize data with genuine long-range dependencies via information-theoretic verification, selecting contexts that reduce a base model's predictive entropy. However, their single-pass offline construction with a fixed model creates a fundamental off-policy gap: the static screening landscape misaligns with the model's evolving capabilities, causing the training distribution to drift. We propose PolicyLong, shifting data construction towards a dynamic on-policy paradigm. By iteratively re-executing data screening (entropy computation, retrieval, and verification) using the current model, PolicyLong ensures the training distribution tracks evolving capabilities, yielding an emergent self-curriculum. Crucially, both positive and hard negative contexts derive from the current model's entropy landscape, co-evolving what the model learns to exploit and resist. Experiments on RULER, HELMET, and LongBench-v2 (Qwen2.5-3B) show PolicyLong consistently outperforms EntropyLong and NExtLong, with gains growing at longer contexts (e.g., +2.54 at 128K on RULER), confirming the value of on-policy data evolution.