Beyond Fixed Length: Bucket Pre-training is All You Need

TL;DR

This paper introduces a multi-bucket data composition method for LLM pre-training that overcomes fixed-length limitations, improving efficiency and effectiveness by adaptively organizing training data based on new quality metrics.

Contribution

It proposes a novel multi-bucket data composition approach with quantitative metrics, enhancing pre-training flexibility and performance of large language models.

Findings

Significant improvements in pre-training efficiency.

Enhanced model performance on downstream tasks.

Effective data organization beyond fixed-length constraints.

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance across various tasks, with pre-training stage serving as the cornerstone of their capabilities. However, the conventional fixed-length data composition strategy for pre-training presents several practical challenges. When using shorter sequences, documents are often truncated, potentially leading to information loss and affecting the model's ability to capture long-range dependencies. Conversely, longer sequences require concatenation of multiple documents, which can introduce noise and affect the natural document boundaries and semantic coherence as well as require substantial computational overhead. To address these challenges, we first establish three quantitative metrics for evaluating data composition quality: padding ratio, truncation ratio, and concatenation ratio. Building upon these metrics, we propose a novel multi-bucket data composition method that transcends the fixed-length paradigm. Our approach adaptively organizes training data to achieve optimal composition quality as measured by the proposed metrics, offering a more flexible and efficient approach for pre-training. We conduct extensive experiments and the results demonstrate that our proposed method significantly enhances both the efficiency and effectiveness of LLM pre-training.