Elixir: Train a Large Language Model on a Small GPU Cluster

TL;DR

Elixir is an automated system that optimizes large language model training on small GPU clusters by intelligently combining partitioning and offloading techniques, significantly improving training speed without requiring expert tuning.

Contribution

Elixir introduces an automated, profiling-based approach to optimize large-model training on limited hardware, surpassing existing manual tuning methods.

Findings

Achieves up to 3.4× speedup on GPT-2 models

Outperforms current state-of-the-art baseline methods

Automates the optimization process for resource-constrained training

Abstract

In recent years, large language models have achieved great success due to their unprecedented size. However, training these models poses a challenge for most researchers as it requires a substantial number of GPUs. To reduce GPU memory usage, memory partitioning, and memory offloading have been proposed. These approaches eliminate memory redundancies and offload memory usage to the CPU and NVMe memory, respectively, enabling training on small GPU clusters. However, directly deploying these solutions often leads to suboptimal efficiency. Only experienced experts can unleash the full potential of hardware by carefully tuning the distributed configuration. Thus, we present a novel solution, Elixir, which automates efficient large-model training based on pre-runtime model profiling. Elixir aims to identify the optimal combination of partitioning and offloading techniques to maximize training throughput. In our experiments, Elixir significantly outperforms the current state-of-the-art baseline. Our optimal configuration achieves up to a 3.4$\times$ speedup on GPT-2 models compared with SOTA solutions. We hope that our work will benefit individuals who lack computing resources and expertise, granting them access to large models. The beta version of Elixir is now available at https://github.com/hpcaitech/ColossalAI/tree/feature/elixir.