How to Attain Communication-Efficient DNN Training? Convert, Compress, Correct

TL;DR

This paper presents CO3, a novel algorithm for communication-efficient federated DNN training that combines gradient quantization, lossless compression, and error correction to reduce communication load while maintaining training performance.

Contribution

The paper introduces CO3, a new gradient compression scheme with a rigorous convergence analysis and validated assumptions, improving communication efficiency in federated DNN training.

Findings

CO3 reduces communication load effectively.

Validated the normal distribution assumption for gradients.

Outperforms existing gradient compression methods.

Abstract

This paper introduces CO3 -- an algorithm for communication-efficient federated Deep Neural Network (DNN) training. CO3 takes its name from three processing applied which reduce the communication load when transmitting the local DNN gradients from the remote users to the Parameter Server. Namely: (i) gradient quantization through floating-point conversion, (ii) lossless compression of the quantized gradient, and (iii) quantization error correction. We carefully design each of the steps above to assure good training performance under a constraint on the communication rate. In particular, in steps (i) and (ii), we adopt the assumption that DNN gradients are distributed according to a generalized normal distribution, which is validated numerically in the paper. For step (iii), we utilize an error feedback with memory decay mechanism to correct the quantization error introduced in step (i). We argue that the memory decay coefficient, similarly to the learning rate, can be optimally tuned to improve convergence. A rigorous convergence analysis of the proposed CO3 with SGD is provided. Moreover, with extensive simulations, we show that CO3 offers improved performance when compared with existing gradient compression schemes in the literature which employ sketching and non-uniform quantization of the local gradients.