Resilient Binary Neural Network

TL;DR

This paper introduces ReBNN, a resilient training method for binary neural networks that mitigates weight oscillation by parameterizing the scaling factor and using an adaptive loss, leading to improved performance across vision and NLP tasks.

Contribution

The paper proposes a novel approach to control weight oscillation in BNNs by parameterizing the scaling factor and incorporating a weighted reconstruction loss, with theoretical and empirical validation.

Findings

ReBNN achieves 66.9% Top-1 accuracy on ImageNet with ResNet-18.

ReBNN outperforms prior BNN methods across multiple models and tasks.

The method effectively mitigates weight oscillation during training.

Abstract

Binary neural networks (BNNs) have received ever-increasing popularity for their great capability of reducing storage burden as well as quickening inference time. However, there is a severe performance drop compared with real-valued networks, due to its intrinsic frequent weight oscillation during training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN) to mitigate the frequent oscillation for better BNNs' training. We identify that the weight oscillation mainly stems from the non-parametric scaling factor. To address this issue, we propose to parameterize the scaling factor and introduce a weighted reconstruction loss to build an adaptive training objective. For the first time, we show that the weight oscillation is controlled by the balanced parameter attached to the reconstruction loss, which provides a theoretical foundation to parameterize it in back propagation. Based on this, we learn our ReBNN by calculating the balanced parameter based on its maximum magnitude, which can effectively mitigate the weight oscillation with a resilient training process. Extensive experiments are conducted upon various network models, such as ResNet and Faster-RCNN for computer vision, as well as BERT for natural language processing. The results demonstrate the overwhelming performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9% Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing existing state-of-the-arts by a significant margin. Our code is open-sourced at https://github.com/SteveTsui/ReBNN.