From promise to practice: realizing high-performance decentralized training

TL;DR

This paper explores how decentralized training can outperform traditional methods like All-Reduce in multi-node deep learning, by analyzing key factors, proposing a new Adam algorithm, and demonstrating practical benefits on GPU clusters.

Contribution

It identifies critical factors for speedups in decentralized training, develops a runtime model, and introduces a convergent decentralized Adam algorithm with variance mitigation techniques.

Findings

Decentralized training can outperform All-Reduce in certain conditions.

The proposed decentralized Adam algorithm converges and supports overlapping communication and computation.

Experimental results show improved runtime and generalization on GPU clusters.

Abstract

Decentralized training of deep neural networks has attracted significant attention for its theoretically superior scalability over synchronous data-parallel methods like All-Reduce. However, realizing this potential in multi-node training is challenging due to the complex design space that involves communication topologies, computation patterns, and optimization algorithms. This paper identifies three key factors that can lead to speedups over All-Reduce training and constructs a runtime model to determine when, how, and to what degree decentralization can yield shorter per-iteration runtimes. Furthermore, to support the decentralized training of transformer-based models, we study a decentralized Adam algorithm that allows for overlapping communications and computations, prove its convergence, and propose an accumulation technique to mitigate the high variance caused by small local batch sizes. We deploy the proposed approach in clusters with up to 64 GPUs and demonstrate its practicality and advantages in both runtime and generalization performance under a fixed iteration budget.