Communication Compression for Decentralized Training

TL;DR

This paper introduces a framework for communication compression in decentralized training, combining techniques to handle both low bandwidth and high latency networks, and proposes algorithms with proven convergence rates.

Contribution

It develops a novel framework and algorithms for compressed decentralized training that effectively address both bandwidth and latency challenges, with theoretical convergence guarantees.

Findings

Algorithms converge at rate O(1/√nT) matching centralized training.

Proposed methods outperform existing decentralized and quantized algorithms.

Effective in networks with both high latency and low bandwidth.

Abstract

Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: {\em communication compression} for low bandwidth networks, and {\em decentralization} for high latency networks. In this paper, We explore a natural question: {\em can the combination of both techniques lead to a system that is robust to both bandwidth and latency?} Although the system implication of such combination is trivial, the underlying theoretical principle and algorithm design is challenging: unlike centralized algorithms, simply compressing exchanged information, even in an unbiased stochastic way, within the decentralized network would accumulate the error and fail to converge. In this paper, we develop a framework of compressed, decentralized training and propose two different strategies, which we call {\em extrapolation compression} and {\em difference compression}. We analyze both algorithms and prove both converge at the rate of $O(1/\sqrt{nT})$ where $n$ is the number of workers and $T$ is the number of iterations, matching the convergence rate for full precision, centralized training. We validate our algorithms and find that our proposed algorithm outperforms the best of merely decentralized and merely quantized algorithm significantly for networks with {\em both} high latency and low bandwidth.