Coded Deep Learning: Framework and Algorithm

TL;DR

This paper introduces coded deep learning (CDL), a framework that integrates information-theoretic coding into deep learning to compress models, reduce computational complexity, and enable efficient parallelism, with empirical results outperforming existing methods.

Contribution

The paper presents a novel framework called CDL that combines probabilistic quantization and information-theoretic constraints to compress deep learning models and improve training and inference efficiency.

Findings

CDL outperforms state-of-the-art DNN compression algorithms.

R-CDL achieves a better accuracy-compression trade-off.

Significant reduction in training and inference complexity.

Abstract

The success of deep learning (DL) is often achieved with large models and high complexity during both training and post-training inferences, hindering training in resource-limited settings. To alleviate these issues, this paper introduces a new framework dubbed ``coded deep learning'' (CDL), which integrates information-theoretic coding concepts into the inner workings of DL, to significantly compress model weights and activations, reduce computational complexity at both training and post-training inference stages, and enable efficient model/data parallelism. Specifically, within CDL, (i) we first propose a novel probabilistic method for quantizing both model weights and activations, and its soft differentiable variant which offers an analytic formula for gradient calculation during training; (ii) both the forward and backward passes during training are executed over quantized weights and activations, eliminating most floating-point operations and reducing training complexity; (iii) during training, both weights and activations are entropy constrained so that they are compressible in an information-theoretic sense throughout training, thus reducing communication costs in model/data parallelism; and (iv) the trained model in CDL is by default in a quantized format with compressible quantized weights, reducing post-training inference and storage complexity. Additionally, a variant of CDL, namely relaxed CDL (R-CDL), is presented to further improve the trade-off between validation accuracy and compression though requiring full precision in training with other advantageous features of CDL intact. Extensive empirical results show that CDL and R-CDL outperform the state-of-the-art algorithms in DNN compression in the literature.