Refining Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models

TL;DR

This paper compares packing strategies for training language models, finding that matching atom size to maximum sequence length optimizes performance, with padding offering lower perplexity but less efficiency than concatenation.

Contribution

It systematically evaluates the impact of atom size and packing strategies on language model training performance and efficiency.

Findings

Matching atom size to MSL optimizes model performance.

Padding packing yields lower perplexity than concatenation.

Padding requires more training steps and reduces compute efficiency.

Abstract

Packing and shuffling tokens is a common practice in training auto-regressive language models (LMs) to prevent overfitting and improve efficiency. Typically documents are concatenated to chunks of maximum sequence length (MSL) and then shuffled. However setting the atom size, the length for each data chunk accompanied by random shuffling, to MSL may lead to contextual incoherence due to tokens from different documents being packed into the same chunk. An alternative approach is to utilize padding, another common data packing strategy, to avoid contextual incoherence by only including one document in each shuffled chunk. To optimize both packing strategies (concatenation vs padding), we investigated the optimal atom size for shuffling and compared their performance and efficiency. We found that matching atom size to MSL optimizes performance for both packing methods (concatenation and padding), and padding yields lower final perplexity (higher performance) than concatenation at the cost of more training steps and lower compute efficiency. This trade-off informs the choice of packing methods in training language models.