Bayesian Compression for Deep Learning

TL;DR

This paper proposes a Bayesian approach to deep learning compression, using hierarchical priors for node pruning and posterior uncertainties for optimal weight encoding, achieving state-of-the-art results.

Contribution

Introduces hierarchical priors for node pruning and utilizes posterior uncertainties to optimize weight precision in Bayesian deep learning compression.

Findings

Achieved state-of-the-art compression rates

Effective pruning of network nodes

Optimized fixed point precision based on uncertainty

Abstract

Compression and computational efficiency in deep learning have become a problem of great significance. In this work, we argue that the most principled and effective way to attack this problem is by adopting a Bayesian point of view, where through sparsity inducing priors we prune large parts of the network. We introduce two novelties in this paper: 1) we use hierarchical priors to prune nodes instead of individual weights, and 2) we use the posterior uncertainties to determine the optimal fixed point precision to encode the weights. Both factors significantly contribute to achieving the state of the art in terms of compression rates, while still staying competitive with methods designed to optimize for speed or energy efficiency.