Training Binary Neural Networks with Real-to-Binary Convolutions

TL;DR

This paper introduces a method to train binary neural networks that nearly match the accuracy of full-precision models by minimizing discrepancies between binary and real-valued convolutions, achieving state-of-the-art results on ImageNet.

Contribution

The paper proposes a novel approach combining loss functions and data-driven re-scaling to improve binary neural network accuracy, narrowing the gap with real-valued networks.

Findings

Achieves within 3-5% of full-precision accuracy on ImageNet.

Surpasses previous binary network methods by over 5% top-1 accuracy.

Reduces the accuracy gap to less than 3-5% on CIFAR-100 and ImageNet.

Abstract

This paper shows how to train binary networks to within a few percent points ($\sim 3-5 \%$) of the full precision counterpart. We first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at https://github.com/brais-martinez/real2binary.