Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning

TL;DR

This paper provides a comprehensive analysis of packing versus padding in supervised fine-tuning, evaluating efficiency, performance, and practical considerations across various models and datasets.

Contribution

It offers the first extensive comparison of packing and padding in SFT, including benchmarks, practical insights, and open-source tools for future research.

Findings

Packing is more suitable for large models and datasets in SFT.

Packing can improve training efficiency without sacrificing performance.

The effectiveness of packing depends on model size and dataset characteristics.

Abstract

Packing, initially utilized in the pre-training phase, is an optimization technique designed to maximize hardware resource efficiency by combining different training sequences to fit the model's maximum input length. Although it has demonstrated effectiveness during pre-training, there remains a lack of comprehensive analysis for the supervised fine-tuning (SFT) stage on the following points: (1) whether packing can effectively enhance training efficiency while maintaining performance, (2) the suitable size of the model and dataset for fine-tuning with the packing method, and (3) whether packing unrelated or related training samples might cause the model to either excessively disregard or over-rely on the context. In this paper, we perform extensive comparisons between SFT methods using padding and packing, covering SFT datasets ranging from 69K to 1.2M and models from 8B to 70B. This provides the first comprehensive analysis of the advantages and limitations of packing versus padding, as well as practical considerations for implementing packing in various training scenarios. Our analysis covers various benchmarks, including knowledge, reasoning, and coding, as well as GPT-based evaluations, time efficiency, and other fine-tuning parameters. We also open-source our code for fine-tuning and evaluation and provide checkpoints fine-tuned on datasets of different sizes, aiming to advance future research on packing methods. Code is available at: https://github.com/ShuheWang1998/Packing-Analysis?tab=readme-ov-file.