LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models

TL;DR

LongSkywork introduces a training recipe for large language models to efficiently extend context length up to 200,000 tokens, utilizing synthetic data and a specialized training stage to enhance long-context processing capabilities.

Contribution

The paper presents a novel training approach with a long-context SFT stage and synthetic data methods, enabling efficient extension of LLM context length with minimal additional training.

Findings

Synthetic long-context data can outperform human-curated data.

Models achieve perfect accuracy on long-context retrieval benchmarks.

LongSkywork-13B matches performance of leading models like Claude2.1.

Abstract

We introduce LongSkywork, a long-context Large Language Model (LLM) capable of processing up to 200,000 tokens. We provide a training recipe for efficiently extending context length of LLMs. We identify that the critical element in enhancing long-context processing capability is to incorporate a long-context SFT stage following the standard SFT stage. A mere 200 iterations can convert the standard SFT model into a long-context model. To reduce the effort in collecting and annotating data for long-context language modeling, we develop two novel methods for creating synthetic data. These methods are applied during the continual pretraining phase as well as the Supervised Fine-Tuning (SFT) phase, greatly enhancing the training efficiency of our long-context LLMs. Our findings suggest that synthetic long-context SFT data can surpass the performance of data curated by humans to some extent. LongSkywork achieves outstanding performance on a variety of long-context benchmarks. In the Needle test, a benchmark for long-context information retrieval, our models achieved perfect accuracy across multiple context spans. Moreover, in realistic application scenarios, LongSkywork-13B demonstrates performance on par with Claude2.1, the leading long-context model, underscoring the effectiveness of our proposed methods.