Bimodal Distributed Binarized Neural Networks

TL;DR

This paper introduces a novel binarization method for neural networks called Bi-Modal Distributed Binarization, which uses kurtosis regularization and a training scheme to improve performance and reduce error in binary neural networks.

Contribution

It proposes a new binarization technique that enforces a bi-modal weight distribution using kurtosis regularization and a training scheme called WDM, enhancing binary neural network accuracy.

Findings

Outperforms state-of-the-art on CIFAR-10 and ImageNet

Reduces generalization error in BNNs

Creates robust binary feature maps

Abstract

Binary Neural Networks (BNNs) are an extremely promising method to reduce deep neural networks' complexity and power consumption massively. Binarization techniques, however, suffer from ineligible performance degradation compared to their full-precision counterparts. Prior work mainly focused on strategies for sign function approximation during forward and backward phases to reduce the quantization error during the binarization process. In this work, we propose a Bi-Modal Distributed binarization method (\methodname{}). That imposes bi-modal distribution of the network weights by kurtosis regularization. The proposed method consists of a training scheme that we call Weight Distribution Mimicking (WDM), which efficiently imitates the full-precision network weight distribution to their binary counterpart. Preserving this distribution during binarization-aware training creates robust and informative binary feature maps and significantly reduces the generalization error of the BNN. Extensive evaluations on CIFAR-10 and ImageNet demonstrate the superiority of our method over current state-of-the-art schemes. Our source code, experimental settings, training logs, and binary models are available at \url{https://github.com/BlueAnon/BD-BNN}.