Self-Distribution Binary Neural Networks

TL;DR

This paper introduces Self-Distribution Binary Neural Networks (SD-BNN), which adaptively adjust the sign distribution of features and weights to improve accuracy and reduce computational complexity in binary neural networks.

Contribution

The paper proposes SD-BNN, a novel method that uses activation and weight self-distribution to enhance binary neural networks without relying on scaling factors.

Findings

Outperforms state-of-the-art BNNs on CIFAR-10 and ImageNet

Achieves 92.5% accuracy on CIFAR-10 with ResNet-18

Achieves 66.5% accuracy on ImageNet with ResNet-18

Abstract

In this work, we study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation). Feature representation is critical for deep neural networks, while in BNNs, the features only differ in signs. Prior work introduces scaling factors into binary weights and activations to reduce the quantization error and effectively improves the classification accuracy of BNNs. However, the scaling factors not only increase the computational complexity of networks, but also make no sense to the signs of binary features. To this end, Self-Distribution Binary Neural Network (SD-BNN) is proposed. Firstly, we utilize Activation Self Distribution (ASD) to adaptively adjust the sign distribution of activations, thereby improve the sign differences of the outputs of the convolution. Secondly, we adjust the sign distribution of weights through Weight Self Distribution (WSD) and then fine-tune the sign distribution of the outputs of the convolution. Extensive experiments on CIFAR-10 and ImageNet datasets with various network structures show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs (e.g., achieves 92.5% on CIFAR-10 and 66.5% on ImageNet with ResNet-18) with less computation cost. Code is available at https://github.com/ pingxue-hfut/SD-BNN.