On the Convergence of Memory-Based Distributed SGD

TL;DR

This paper develops a universal convergence analysis for memory-based distributed SGD with momentum, providing theoretical guarantees for both convex and non-convex problems and bridging the gap between theory and practice.

Contribution

It introduces a transformation equation linking M-DSGD to traditional DSGD, enabling convergence analysis with momentum and stagewise learning strategies.

Findings

Provides convergence rates for M-DSGD with momentum in convex and non-convex settings.

Introduces a transformation equation to relate M-DSGD to DSGD for theoretical analysis.

Bridges the gap between theoretical convergence and practical implementations.

Abstract

Distributed stochastic gradient descent~(DSGD) has been widely used for optimizing large-scale machine learning models, including both convex and non-convex models. With the rapid growth of model size, huge communication cost has been the bottleneck of traditional DSGD. Recently, many communication compression methods have been proposed. Memory-based distributed stochastic gradient descent~(M-DSGD) is one of the efficient methods since each worker communicates a sparse vector in each iteration so that the communication cost is small. Recent works propose the convergence rate of M-DSGD when it adopts vanilla SGD. However, there is still a lack of convergence theory for M-DSGD when it adopts momentum SGD. In this paper, we propose a universal convergence analysis for M-DSGD by introducing \emph{transformation equation}. The transformation equation describes the relation between traditional DSGD and M-DSGD so that we can transform M-DSGD to its corresponding DSGD. Hence we get the convergence rate of M-DSGD with momentum for both convex and non-convex problems. Furthermore, we combine M-DSGD and stagewise learning that the learning rate of M-DSGD in each stage is a constant and is decreased by stage, instead of iteration. Using the transformation equation, we propose the convergence rate of stagewise M-DSGD which bridges the gap between theory and practice.