SSDTrain: An Activation Offloading Framework to SSDs for Faster Large Language Model Training

TL;DR

SSDTrain is an adaptive framework that offloads activations to SSDs during large language model training, significantly reducing GPU memory usage while maintaining training speed and compatibility with popular frameworks.

Contribution

It introduces a novel activation offloading method to SSDs that overlaps data transfer with computation, enabling larger models and micro-batches without performance loss.

Findings

Reduces activation peak memory by 47%.

Maintains negligible overhead during training.

Enables larger micro-batches and improved throughput.

Abstract

The growth rate of the GPU memory capacity has not been able to keep up with that of the size of large language models (LLMs), hindering the model training process. In particular, activations -- the intermediate tensors produced during forward propagation and reused in backward propagation -- dominate the GPU memory use. This leads to high training overhead such as high weight update cost due to the small micro-batch size. To address this challenge, we propose SSDTrain, an adaptive activation offloading framework to high-capacity NVMe SSDs. SSDTrain reduces GPU memory usage without impacting performance by fully overlapping data transfers with computation. SSDTrain is compatible with popular deep learning frameworks like PyTorch, Megatron, and DeepSpeed, and it employs techniques such as tensor deduplication and forwarding to further enhance efficiency. We extensively experimented with popular LLMs like GPT, BERT, and T5. Results demonstrate that SSDTrain reduces 47% of the activation peak memory usage. Meanwhile, SSDTrain perfectly overlaps the I/O with the computation and incurs negligible overhead. Compared with keeping activations in GPU memory and layerwise full recomputation, SSDTrain achieves the best memory savings with negligible throughput loss. We further analyze how the reduced activation memory use may be leveraged to increase throughput by increasing micro-batch size and reducing pipeline parallelism bubbles.