Boosting Binary Neural Networks via Dynamic Thresholds Learning

TL;DR

This paper introduces DySign, a dynamic threshold learning method for binarizing neural networks, which adaptively generates thresholds based on input data to improve accuracy in BNNs for vision tasks.

Contribution

The paper proposes DySign, a novel threshold learning approach that enhances BNNs by incorporating input distribution statistics, applicable to both DCNNs and ViTs, significantly improving their performance.

Findings

DyBCNNs outperform baselines by 1.5-1.8% on ImageNet.

DyBinaryCCT achieves nearly 9% higher accuracy than baseline.

The method reduces information loss in binarization.

Abstract

Developing lightweight Deep Convolutional Neural Networks (DCNNs) and Vision Transformers (ViTs) has become one of the focuses in vision research since the low computational cost is essential for deploying vision models on edge devices. Recently, researchers have explored highly computational efficient Binary Neural Networks (BNNs) by binarizing weights and activations of Full-precision Neural Networks. However, the binarization process leads to an enormous accuracy gap between BNN and its full-precision version. One of the primary reasons is that the Sign function with predefined or learned static thresholds limits the representation capacity of binarized architectures since single-threshold binarization fails to utilize activation distributions. To overcome this issue, we introduce the statistics of channel information into explicit thresholds learning for the Sign Function dubbed DySign to generate various thresholds based on input distribution. Our DySign is a straightforward method to reduce information loss and boost the representative capacity of BNNs, which can be flexibly applied to both DCNNs and ViTs (i.e., DyBCNN and DyBinaryCCT) to achieve promising performance improvement. As shown in our extensive experiments. For DCNNs, DyBCNNs based on two backbones (MobileNetV1 and ResNet18) achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset, outperforming baselines by a large margin (i.e., 1.8% and 1.5% respectively). For ViTs, DyBinaryCCT presents the superiority of the convolutional embedding layer in fully binarized ViTs and achieves 56.1% on the ImageNet dataset, which is nearly 9% higher than the baseline.