Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

TL;DR

Alpa automates the complex process of parallelizing large deep learning models across distributed devices by automatically generating hierarchical execution plans, improving scalability and efficiency over manual or limited automatic methods.

Contribution

Alpa introduces a hierarchical parallelism framework and compilation techniques to automatically generate efficient distributed training plans for large models.

Findings

Matches or outperforms hand-tuned systems

Generalizes to heterogeneous architectures

Automates complex parallelization planning

Abstract

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations. They do not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive efficient parallel execution plans at each parallelism level. Alpa implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans. Alpa's source code is publicly available at https://github.com/alpa-projects/alpa