Decentralized Deep Learning with Arbitrary Communication Compression

TL;DR

This paper introduces Choco-SGD, a communication-efficient decentralized training algorithm that supports arbitrary high compression ratios, enabling scalable deep learning across distributed devices and data centers with improved speed.

Contribution

It extends Choco-SGD to non-convex functions, proving convergence under high compression ratios and demonstrating practical efficiency in distributed deep learning scenarios.

Findings

Supports arbitrary high compression ratios

Achieves linear speedup with the number of workers

Outperforms all-reduce in practical training scenarios

Abstract

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD $-$ recently introduced and analyzed for strongly-convex objectives only $-$ converges under arbitrary high compression ratio on general non-convex functions at the rate $O\bigl(1/\sqrt{nT}\bigr)$ where $T$ denotes the number of iterations and $n$ the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over distributed user devices, connected by a social network and (ii) in a datacenter (outperforming all-reduce time-wise).