Training Long-Context LLMs Efficiently via Chunk-wise Optimization

TL;DR

This paper introduces SeCO and SpaCO, memory-efficient training methods that enable long-context LLMs to be trained faster and with longer sequences, reducing costs and expanding practical usability.

Contribution

The paper proposes two novel chunk-wise optimization techniques, SeCO and SpaCO, that significantly improve training efficiency and sequence length handling for long-context LLMs.

Findings

SeCO increases maximum sequence length from 1K to 16K tokens.

SpaCO achieves up to 3x faster training speed than SeCO.

Both methods enable training long-context models on limited hardware.

Abstract

While long-context large language models (LLMs) exhibit remarkable document processing capabilities, their prohibitively high training costs often hinder customized applications. To mitigate this issue, we propose \textit{Sequential Chunk-wise Optimization} (SeCO), a memory-efficient training paradigm that partitions lengthy inputs into manageable chunks. Each chunk independently constructs its computational graph and performs localized backpropagation, ensuring that only one chunk's forward activations are stored in memory. Building on SeCO, we further introduce \textit{Sparse Chunk-wise Optimization} (SpaCO), which reduces computational overhead by selectively propagating gradients to specific chunks and incorporates a carefully designed compensation factor to ensure unbiased gradient estimation. SpaCO decouples the computational cost of backpropagation from the context length, enabling training time to gradually converge to inference time as sequences become longer. Implemented as lightweight training wrappers, both SeCO and SpaCO offer substantial practical benefits. For example, when fine-tuning an 8B model with LoRA on a single RTX 3090 GPU, SeCO expands maximum sequence length from 1K to 16K tokens, while SpaCO demonstrates accelerated training speed -- achieving up to 3x faster than SeCO under the same experimental setup. These innovations provide new insights into optimizing long-context models, making them more accessible for practical applications. We have open-sourced the code at \href{https://github.com/wenhaoli-xmu/seco}{here}.