Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

TL;DR

This paper introduces SASG, a novel distributed learning algorithm that reduces communication costs by using sparse and adaptive gradient aggregation, maintaining convergence rates similar to standard stochastic gradient descent.

Contribution

The paper proposes SASG, a new communication-efficient distributed learning method combining sparse communication and adaptive gradient aggregation with proven convergence.

Findings

Reduces communication overhead significantly compared to previous methods.

Maintains convergence rate comparable to stochastic gradient descent.

Scales well with increasing number of workers.

Abstract

Gradient-based optimization methods implemented on distributed computing architectures are increasingly used to tackle large-scale machine learning applications. A key bottleneck in such distributed systems is the high communication overhead for exchanging information, such as stochastic gradients, between workers. The inherent causes of this bottleneck are the frequent communication rounds and the full model gradient transmission in every round. In this study, we present SASG, a communication-efficient distributed algorithm that enjoys the advantages of sparse communication and adaptive aggregated stochastic gradients. By dynamically determining the workers who need to communicate through an adaptive aggregation rule and sparsifying the transmitted information, the SASG algorithm reduces both the overhead of communication rounds and the number of communication bits in the distributed system. For the theoretical analysis, we introduce an important auxiliary variable and define a new Lyapunov function to prove that the communication-efficient algorithm is convergent. The convergence result is identical to the sublinear rate of stochastic gradient descent, and our result also reveals that SASG scales well with the number of distributed workers. Finally, experiments on training deep neural networks demonstrate that the proposed algorithm can significantly reduce communication overhead compared to previous methods.