Stochastic Distributed Learning with Gradient Quantization and Variance Reduction

TL;DR

This paper introduces new distributed optimization algorithms that combine gradient quantization with variance reduction, achieving faster convergence and linear rates even with compressed updates.

Contribution

It proposes the first methods with linear convergence for quantized gradient updates in distributed learning, improving efficiency over existing schemes.

Findings

Converges in f3((a6 + a6 rac{a9}{n} + a9) f3 \, f0(1/\u03b5)) steps for strongly convex functions.

Achieves linear convergence for finite-sum problems with quantized gradients.

Experimental results show improved efficiency over baseline methods.

Abstract

We consider distributed optimization where the objective function is spread among different devices, each sending incremental model updates to a central server. To alleviate the communication bottleneck, recent work proposed various schemes to compress (e.g.\ quantize or sparsify) the gradients, thereby introducing additional variance $\omega \geq 1$ that might slow down convergence. For strongly convex functions with condition number $\kappa$ distributed among $n$ machines, we (i) give a scheme that converges in $\mathcal{O}((\kappa + \kappa \frac{\omega}{n} + \omega)$ $\log (1/\epsilon))$ steps to a neighborhood of the optimal solution. For objective functions with a finite-sum structure, each worker having less than $m$ components, we (ii) present novel variance reduced schemes that converge in $\mathcal{O}((\kappa + \kappa \frac{\omega}{n} + \omega + m)\log(1/\epsilon))$ steps to arbitrary accuracy $\epsilon > 0$. These are the first methods that achieve linear convergence for arbitrary quantized updates. We also (iii) give analysis for the weakly convex and non-convex cases and (iv) verify in experiments that our novel variance reduced schemes are more efficient than the baselines.