Self-Compression in Bayesian Neural Networks

TL;DR

This paper introduces a novel approach where Bayesian neural networks inherently identify and remove redundant parameters, enabling automatic self-compression without sacrificing accuracy, thus reducing computational and storage costs.

Contribution

The paper presents the concept of self-compression in Bayesian neural networks, leveraging uncertainty propagation to automatically discover and eliminate redundant parameters.

Findings

Networks can be compressed by removing parameters identified through Bayesian uncertainty.

Self-compression retains accuracy while reducing model size.

Bayesian framework facilitates automatic redundancy detection.

Abstract

Machine learning models have achieved human-level performance on various tasks. This success comes at a high cost of computation and storage overhead, which makes machine learning algorithms difficult to deploy on edge devices. Typically, one has to partially sacrifice accuracy in favor of an increased performance quantified in terms of reduced memory usage and energy consumption. Current methods compress the networks by reducing the precision of the parameters or by eliminating redundant ones. In this paper, we propose a new insight into network compression through the Bayesian framework. We show that Bayesian neural networks automatically discover redundancy in model parameters, thus enabling self-compression, which is linked to the propagation of uncertainty through the layers of the network. Our experimental results show that the network architecture can be successfully compressed by deleting parameters identified by the network itself while retaining the same level of accuracy.