Adjacent Leader Decentralized Stochastic Gradient Descent

TL;DR

This paper introduces AL-DSGD, a novel decentralized gradient descent method that enhances model performance and convergence speed in communication-limited environments by adaptive weighting and dynamic communication graphs.

Contribution

The paper proposes AL-DSGD, a new decentralized optimization algorithm that improves convergence and performance through adaptive neighbor influence and dynamic graph communication.

Findings

AL-DSGD accelerates convergence compared to state-of-the-art methods.

AL-DSGD improves test performance in communication-constrained settings.

Theoretical convergence of AL-DSGD is established.

Abstract

This work focuses on the decentralized deep learning optimization framework. We propose Adjacent Leader Decentralized Gradient Descent (AL-DSGD), for improving final model performance, accelerating convergence, and reducing the communication overhead of decentralized deep learning optimizers. AL-DSGD relies on two main ideas. Firstly, to increase the influence of the strongest learners on the learning system it assigns weights to different neighbor workers according to both their performance and the degree when averaging among them, and it applies a corrective force on the workers dictated by both the currently best-performing neighbor and the neighbor with the maximal degree. Secondly, to alleviate the problem of the deterioration of the convergence speed and performance of the nodes with lower degrees, AL-DSGD relies on dynamic communication graphs, which effectively allows the workers to communicate with more nodes while keeping the degrees of the nodes low. Experiments demonstrate that AL-DSGD accelerates the convergence of the decentralized state-of-the-art techniques and improves their test performance especially in the communication constrained environments. We also theoretically prove the convergence of the proposed scheme. Finally, we release to the community a highly general and concise PyTorch-based library for distributed training of deep learning models that supports easy implementation of any distributed deep learning approach ((a)synchronous, (de)centralized).