Learning Discrete Weights and Activations Using the Local Reparameterization Trick

TL;DR

This paper introduces a method to train neural networks with discrete weights and activations using the local reparameterization trick, achieving efficient inference suitable for low-resource devices.

Contribution

It extends previous discrete weight training methods to include discrete activations, improving efficiency and reducing memory use at inference.

Findings

Achieves state-of-the-art results with binary activations

Reduces computational complexity with bitwise operations

Enables deployment on low-resource devices

Abstract

In computer vision and machine learning, a crucial challenge is to lower the computation and memory demands for neural network inference. A commonplace solution to address this challenge is through the use of binarization. By binarizing the network weights and activations, one can significantly reduce computational complexity by substituting the computationally expensive floating operations with faster bitwise operations. This leads to a more efficient neural network inference that can be deployed on low-resource devices. In this work, we extend previous approaches that trained networks with discrete weights using the local reparameterization trick to also allow for discrete activations. The original approach optimized a distribution over the discrete weights and uses the central limit theorem to approximate the pre-activation with a continuous Gaussian distribution. Here we show that the probabilistic modeling can also allow effective training of networks with discrete activation as well. This further reduces runtime and memory footprint at inference time with state-of-the-art results for networks with binary activations.