PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training

TL;DR

PipeFill enhances GPU utilization during pipeline-parallel large language model training by filling pipeline bubbles with auxiliary jobs, significantly increasing efficiency with minimal slowdown.

Contribution

This paper introduces PipeFill, a novel method to fill pipeline bubbles in GPU training, improving utilization and scalability of large-scale LLM training.

Findings

GPU utilization increased by up to 63%

Training slowdown kept below 2%

Additional work equivalent to 2,600 GPUs at 8K GPU scale

Abstract

Training Deep Neural Networks (DNNs) with billions of parameters generally involves pipeline-parallel (PP) execution. Unfortunately, PP model training can use GPUs inefficiently, especially at large scale, due to idle GPU time caused by pipeline bubbles, which are often 15-30% and can exceed 60% of the training job's GPU allocation. To improve the GPU utilization of PP model training, this paper describes PipeFill, which fills pipeline bubbles with execution of other pending jobs. By leveraging bubble GPU time, PipeFill reduces the GPU utilization sacrifice associated with scaling-up of large-model training. To context-switch between fill jobs and the main training job with minimal overhead to the main job, and maximize fill job efficiency, PipeFill carefully fits fill job work to measured bubble durations and GPU memory availability, introduces explicit pipeline-bubble instructions, and orchestrates placement and execution of fill jobs in pipeline bubbles. Experiments show that PipeFill can increase overall utilization by up to 63% for GPUs used in large-scale LLM training, with <2% slowdown of the training job, and 5-15% even for low-scale LLM training. For large-scale LLM training on 8K GPUs, the 63% increase translates to up to 2.6K additional GPUs worth of work completed.