Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

TL;DR

Colossal-AI is a comprehensive deep learning system that simplifies large-scale model training by supporting multiple parallel strategies, significantly improving training speed and efficiency.

Contribution

It introduces a unified interface for various parallel training methods, enabling scalable and efficient training of billion-parameter models without requiring extensive domain expertise.

Findings

Achieves up to 2.76x training speedup on large models

Supports multiple parallelism strategies within a unified system

Reduces the need for domain expertise in parallel training

Abstract

The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice for choosing the optimal parallel strategy is still lacking, since it requires domain expertise in both deep learning and parallel computing. The Colossal-AI system addressed the above challenge by introducing a unified interface to scale your sequential code of model training to distributed environments. It supports parallel training methods such as data, pipeline, tensor, and sequence parallelism, as well as heterogeneous training methods integrated with zero redundancy optimizer. Compared to the baseline system, Colossal-AI can achieve up to 2.76 times training speedup on large-scale models.