Bit-wise Training of Neural Network Weights

TL;DR

This paper presents a novel bit-wise training algorithm for neural network weights that enables learning at the bit level, resulting in sparse, efficient, and versatile networks with comparable accuracy to traditional methods.

Contribution

It introduces a new bit-wise training algorithm for neural networks that naturally uncovers sparsity and allows for arbitrary bit-depth weight representation without extra regularization.

Findings

Training bits in the most significant positions yields the highest accuracy.

Networks can store arbitrary codes without impacting performance.

The method outperforms standard training in fully connected networks.

Abstract

We introduce an algorithm where the individual bits representing the weights of a neural network are learned. This method allows training weights with integer values on arbitrary bit-depths and naturally uncovers sparse networks, without additional constraints or regularization techniques. We show better results than the standard training technique with fully connected networks and similar performance as compared to standard training for convolutional and residual networks. By training bits in a selective manner we found that the biggest contribution to achieving high accuracy is given by the first three most significant bits, while the rest provide an intrinsic regularization. As a consequence more than 90\% of a network can be used to store arbitrary codes without affecting its accuracy. These codes may be random noise, binary files or even the weights of previously trained networks.