ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions

TL;DR

ReActNet introduces generalized activation functions and a distributional loss to significantly improve the accuracy of binary neural networks, narrowing the gap with real-valued models without extra computational cost.

Contribution

It proposes a novel baseline binary network with parameter-free shortcuts and generalized activation functions, achieving state-of-the-art accuracy on ImageNet.

Findings

Outperforms existing binary networks by 3.6-4.0% in top-1 accuracy.

Reduces the accuracy gap to real-valued networks to within 3%.

Achieves superior efficiency and accuracy trade-offs.

Abstract

In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts, bypassing all the intermediate convolutional layers including the downsampling layers. This baseline network strikes a good trade-off between accuracy and efficiency, achieving superior performance than most of existing binary networks at approximately half of the computational cost. Through extensive experiments and analysis, we observed that the performance of binary networks is sensitive to activation distribution variations. Based on this important observation, we propose to generalize the traditional Sign and PReLU functions, denoted as RSign and RPReLU for the respective generalized functions, to enable explicit learning of the distribution reshape and shift at near-zero extra cost. Lastly, we adopt a distributional loss to further enforce the binary network to learn similar output distributions as those of a real-valued network. We show that after incorporating all these ideas, the proposed ReActNet outperforms all the state-of-the-arts by a large margin. Specifically, it outperforms Real-to-Binary Net and MeliusNet29 by 4.0% and 3.6% respectively for the top-1 accuracy and also reduces the gap to its real-valued counterpart to within 3.0% top-1 accuracy on ImageNet dataset. Code and models are available at: https://github.com/liuzechun/ReActNet.