Distributed learning with compressed gradients

TL;DR

This paper develops a unified theoretical framework for analyzing distributed gradient methods that use asynchronous updates and gradient compression, providing explicit convergence bounds and demonstrating fast convergence with limited communication.

Contribution

It introduces a comprehensive analysis framework for distributed gradient methods with staled and compressed gradients, deriving explicit convergence bounds and characterizing the effects of asynchrony and compression.

Findings

Explicit convergence bounds for compressed gradient algorithms

Trade-offs between asynchrony, compression accuracy, and convergence speed

Numerical validation of fast convergence with limited communication

Abstract

Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bounds on convergence rates and information exchange are derived for several optimization algorithms. These bounds give explicit expressions for step-sizes and characterize how the amount of asynchrony and the compression accuracy affect iteration and communication complexity guarantees. Numerical results highlight convergence properties of different gradient compression algorithms and confirm that fast convergence under limited information exchange is indeed possible.