Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

TL;DR

This paper introduces dataset decomposition, a variable sequence length training method for large language models that reduces computational costs and improves performance by sampling from buckets of different sequence lengths with a curriculum.

Contribution

We propose a novel dataset decomposition technique that enables efficient training of LLMs with variable sequence lengths, outperforming the traditional concat-and-chunk approach.

Findings

Training with dataset decomposition is up to 6x faster than baseline.

Our method achieves comparable or better accuracy on language and long-context benchmarks.

Efficient long-sequence training scales well with dataset size.

Abstract

Large language models (LLMs) are commonly trained on datasets consisting of fixed-length token sequences. These datasets are created by randomly concatenating documents of various lengths and then chunking them into sequences of a predetermined target length (concat-and-chunk). Recent attention implementations mask cross-document attention, reducing the effective length of a chunk of tokens. Additionally, training on long sequences becomes computationally prohibitive due to the quadratic cost of attention. In this study, we introduce dataset decomposition, a novel variable sequence length training technique, to tackle these challenges. We decompose a dataset into a union of buckets, each containing sequences of the same size extracted from a unique document. During training, we use variable sequence length and batch-size, sampling simultaneously from all buckets with a curriculum. In contrast to the concat-and-chunk baseline, which incurs a fixed attention cost at every step of training, our proposed method incurs a computational cost proportional to the actual document lengths at each step, resulting in significant savings in training time. We train an 8k context-length 1B model at the same cost as a 2k context-length model trained with the baseline approach. Experiments on a web-scale corpus demonstrate that our approach significantly enhances performance on standard language evaluations and long-context benchmarks, reaching target accuracy with up to 6x faster training compared to the baseline. Our method not only enables efficient pretraining on long sequences but also scales effectively with dataset size. Lastly, we shed light on a critical yet less studied aspect of training large language models: the distribution and curriculum of sequence lengths, which results in a non-negligible difference in performance.