Probabilistic Binary Neural Networks

TL;DR

This paper introduces BLRNet, a probabilistic training approach for binary neural networks that improves training stability, enables ensemble predictions, and maintains efficiency at test time.

Contribution

It presents a novel probabilistic training method for binary neural networks that avoids non-differentiable gradient approximations and introduces stochastic normalization techniques.

Findings

BLRNet achieves competitive accuracy on benchmarks.

Ensemble predictions improve performance and uncertainty estimation.

Stochastic Batch Normalization and max pooling transfer well to deterministic networks.

Abstract

Low bit-width weights and activations are an effective way of combating the increasing need for both memory and compute power of Deep Neural Networks. In this work, we present a probabilistic training method for Neural Network with both binary weights and activations, called BLRNet. By embracing stochasticity during training, we circumvent the need to approximate the gradient of non-differentiable functions such as sign(), while still obtaining a fully Binary Neural Network at test time. Moreover, it allows for anytime ensemble predictions for improved performance and uncertainty estimates by sampling from the weight distribution. Since all operations in a layer of the BLRNet operate on random variables, we introduce stochastic versions of Batch Normalization and max pooling, which transfer well to a deterministic network at test time. We evaluate the BLRNet on multiple standardized benchmarks.