Convert, compress, correct: Three steps toward communication-efficient DNN training

TL;DR

This paper presents $ ext{CO}_3$, a three-step algorithm combining quantization, compression, and error correction to improve communication efficiency in distributed DNN training over constrained links.

Contribution

The paper introduces $ ext{CO}_3$, a novel joint protocol that effectively balances gradient quantization, compression, and error correction for efficient distributed training.

Findings

Demonstrates improved training efficiency over CIFAR-10

Balances three gradient processing steps for robustness

Achieves high performance with constrained communication links

Abstract

In this paper, we introduce a novel algorithm, $\mathsf{CO}_3$, for communication-efficiency distributed Deep Neural Network (DNN) training. $\mathsf{CO}_3$ is a joint training/communication protocol, which encompasses three processing steps for the network gradients: (i) quantization through floating-point conversion, (ii) lossless compression, and (iii) error correction. These three components are crucial in the implementation of distributed DNN training over rate-constrained links. The interplay of these three steps in processing the DNN gradients is carefully balanced to yield a robust and high-performance scheme. The performance of the proposed scheme is investigated through numerical evaluations over CIFAR-10.