Unified Compression Algorithm for Distributed Nonconvex Optimization: Generalized to 1-Bit, Saturation, and Bounded Noise

TL;DR

This paper introduces a unified compression algorithm for distributed nonconvex optimization that handles various bounded compressors, providing rigorous convergence analysis and improved rates.

Contribution

It generalizes to multiple bounded compression schemes, offers convergence guarantees, and achieves state-of-the-art rates in distributed nonconvex optimization.

Findings

Achieves an $ ext{O}(1/\sqrt{T})$ convergence rate for locally-bounded compressors.

One initial uncompressed communication round improves the rate to $ ext{O}(1/T^{2/3})$.

Recovers state-of-the-art convergence rates under the Polyak-Lojasiewicz condition.

Abstract

In this paper, we propose a unified compression algorithm for distributed nonconvex opitmization with both the locally- and globally-bounded communication compressors, including 1-bit compressors, saturating quantizers, and the globally-bounded compressors with both relative and absolute compression errors, as well as additional arbitrary bounded noise. We provide a rigorous convergence analysis in nonconvex settings and establish linear convergence under the Polyak-Lojasiewicz (P-L) condition. Notably, we establish an $\mathcal{O}(1/\sqrt{T})$ convergence rate for the locally-bounded class in the distributed nonconvex setting, matching that achieved by the centralized algorithms with 1-bit compressors, where $T$ denotes the total number of iterations. Moreover, one initial uncompressed communication round further yields an order-wise improvement to $\mathcal{O}(1/T^{2/3})$. For the P-L setting and the globally-bounded class, we recover state-of-the-art convergence rates.