Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression

TL;DR

This paper establishes a theoretical lower bound for convergence in distributed learning with communication compression and introduces NEOLITHIC, an algorithm that nearly attains this bound, advancing understanding of communication-efficient distributed algorithms.

Contribution

The paper provides the first lower bound for convergence rates under communication compression and proposes NEOLITHIC, an algorithm that nearly matches this bound in distributed non-convex optimization.

Findings

Lower bound for convergence rates established for compressed communication algorithms.

NEOLITHIC algorithm nearly attains the theoretical lower bound, up to logarithmic factors.

Bidirectional contractive compression can achieve convergence speeds comparable to unidirectional unbiased compression.

Abstract

Recent advances in distributed optimization and learning have shown that communication compression is one of the most effective means of reducing communication. While there have been many results on convergence rates under communication compression, a theoretical lower bound is still missing. Analyses of algorithms with communication compression have attributed convergence to two abstract properties: the unbiased property or the contractive property. They can be applied with either unidirectional compression (only messages from workers to server are compressed) or bidirectional compression. In this paper, we consider distributed stochastic algorithms for minimizing smooth and non-convex objective functions under communication compression. We establish a convergence lower bound for algorithms whether using unbiased or contractive compressors in unidirection or bidirection. To close the gap between the lower bound and the existing upper bounds, we further propose an algorithm, NEOLITHIC, which almost reaches our lower bound (up to logarithm factors) under mild conditions. Our results also show that using contractive bidirectional compression can yield iterative methods that converge as fast as those using unbiased unidirectional compression. The experimental results validate our findings.