Self-Binarizing Networks

TL;DR

This paper introduces a novel training method for self-binarizing neural networks that evolve their weights and activations into binary form during training, improving efficiency and accuracy over existing approaches.

Contribution

The authors propose a smooth activation function that is iteratively sharpened to achieve binarization, along with a simplified binary batch normalization technique, enabling more efficient training of binary networks.

Findings

Achieves higher classification accuracy than state-of-the-art methods.

Reduces memory and computation compared to traditional networks.

Demonstrates effectiveness on multiple benchmark datasets.

Abstract

We present a method to train self-binarizing neural networks, that is, networks that evolve their weights and activations during training to become binary. To obtain similar binary networks, existing methods rely on the sign activation function. This function, however, has no gradients for non-zero values, which makes standard backpropagation impossible. To circumvent the difficulty of training a network relying on the sign activation function, these methods alternate between floating-point and binary representations of the network during training, which is sub-optimal and inefficient. We approach the binarization task by training on a unique representation involving a smooth activation function, which is iteratively sharpened during training until it becomes a binary representation equivalent to the sign activation function. Additionally, we introduce a new technique to perform binary batch normalization that simplifies the conventional batch normalization by transforming it into a simple comparison operation. This is unlike existing methods, which are forced to the retain the conventional floating-point-based batch normalization. Our binary networks, apart from displaying advantages of lower memory and computation as compared to conventional floating-point and binary networks, also show higher classification accuracy than existing state-of-the-art methods on multiple benchmark datasets.