Adaptive Compression for Communication-Efficient Distributed Training

TL;DR

This paper introduces AdaCGD, an adaptive compression algorithm for distributed training that dynamically selects compression levels, improving communication efficiency and convergence rates over existing methods.

Contribution

The paper presents a multi-adaptive compression method that extends the 3PC framework, allowing dynamic selection of compression levels and bidirectional compression, with proven convergence guarantees.

Findings

Superior convergence rates compared to existing adaptive methods.

Effective adaptation to various compression mechanisms like Top-K and quantization.

Theoretical bounds established for convex, strongly convex, and nonconvex settings.

Abstract

We propose Adaptive Compressed Gradient Descent (AdaCGD) - a novel optimization algorithm for communication-efficient training of supervised machine learning models with adaptive compression level. Our approach is inspired by the recently proposed three point compressor (3PC) framework of Richtarik et al. (2022), which includes error feedback (EF21), lazily aggregated gradient (LAG), and their combination as special cases, and offers the current state-of-the-art rates for these methods under weak assumptions. While the above mechanisms offer a fixed compression level, or adapt between two extremes only, our proposal is to perform a much finer adaptation. In particular, we allow the user to choose any number of arbitrarily chosen contractive compression mechanisms, such as Top-K sparsification with a user-defined selection of sparsification levels K, or quantization with a user-defined selection of quantization levels, or their combination. AdaCGD chooses the appropriate compressor and compression level adaptively during the optimization process. Besides i) proposing a theoretically-grounded multi-adaptive communication compression mechanism, we further ii) extend the 3PC framework to bidirectional compression, i.e., we allow the server to compress as well, and iii) provide sharp convergence bounds in the strongly convex, convex and nonconvex settings. The convex regime results are new even for several key special cases of our general mechanism, including 3PC and EF21. In all regimes, our rates are superior compared to all existing adaptive compression methods.