Exponential Graph is Provably Efficient for Decentralized Deep Training

TL;DR

This paper demonstrates that exponential graphs enable efficient decentralized deep training by achieving fast communication and effective averaging, leading to state-of-the-art performance in decentralized SGD with minimal communication overhead.

Contribution

It introduces exponential graphs as a topology for decentralized SGD, proving their efficiency and showing how sequences of one-peer exponential graphs can achieve exact averaging with minimal communication.

Findings

Exponential graphs enable fast communication and effective averaging.

Sequences of one-peer exponential graphs can achieve exact averaging.

Decentralized SGD over exponential graphs achieves state-of-the-art efficiency.

Abstract

Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging. The less exact the averaging is, however, the more the total iterations the training needs to take. Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication. This requires a skillful choice of communication topology, which is an under-studied topic in decentralized optimization. In this paper, we study so-called exponential graphs where every node is connected to $O(\log(n))$ neighbors and $n$ is the total number of nodes. This work proves such graphs can lead to both fast communication and effective averaging simultaneously. We also discover that a sequence of $\log(n)$ one-peer exponential graphs, in which each node communicates to one single neighbor per iteration, can together achieve exact averaging. This favorable property enables one-peer exponential graph to average as effective as its static counterpart but communicates more efficiently. We apply these exponential graphs in decentralized (momentum) SGD to obtain the state-of-the-art balance between per-iteration communication and iteration complexity among all commonly-used topologies. Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and high-quality training. Our code is implemented through BlueFog and available at https://github.com/Bluefog-Lib/NeurIPS2021-Exponential-Graph.