NExtLong: Toward Effective Long-Context Training without Long Documents

TL;DR

NExtLong introduces a novel data synthesis framework for training large language models to better understand long-range dependencies without needing actual long documents, improving performance on long-context benchmarks.

Contribution

NExtLong presents a new method for synthesizing long-context data using negative document extension, enhancing long-range dependency modeling in language models.

Findings

Significant performance improvements on HELMET and RULER benchmarks.

Reduces reliance on non-synthetic long documents for training.

Effective in enhancing long-context understanding in LLMs.

Abstract

Large language models (LLMs) with extended context windows have made significant strides yet remain a challenge due to the scarcity of long documents. Existing methods tend to synthesize long-context data but lack a clear mechanism to reinforce the long-range dependency modeling. To address this limitation, we propose NExtLong, a novel framework for synthesizing long-context data through Negative document Extension. NExtLong decomposes a document into multiple meta-chunks and extends the context by interleaving hard negative distractors retrieved from pretraining corpora. This approach compels the model to discriminate long-range dependent context from distracting content, enhancing its ability to model long-range dependencies. Extensive experiments demonstrate that NExtLong achieves significant performance improvements on the HELMET and RULER benchmarks compared to existing long-context synthesis approaches and leading models, which are trained on non-synthetic long documents. These findings highlight NExtLong's ability to reduce reliance on non-synthetic long documents, making it an effective framework for developing advanced long-context LLMs.