Neural Networks with Few Multiplications

TL;DR

This paper introduces a training method for neural networks that reduces multiplications by stochastically binarizing weights and quantizing representations, leading to faster, hardware-efficient training without sacrificing accuracy.

Contribution

The paper presents a novel approach combining stochastic binarization of weights and quantization of activations to eliminate most multiplications during training.

Findings

Achieves comparable or better accuracy than standard training on MNIST, CIFAR10, SVHN.

Reduces computational complexity, enabling faster, hardware-friendly neural network training.

Demonstrates effectiveness across multiple datasets with minimal performance loss.

Abstract

For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.