BD-Net: Has Depth-Wise Convolution Ever Been Applied in Binary Neural Networks?

TL;DR

This paper introduces a novel approach to binarize depth-wise convolutions in Binary Neural Networks, significantly improving efficiency and accuracy across multiple datasets by combining a 1.58-bit convolution and residual stabilization techniques.

Contribution

It presents the first successful binarization of depth-wise convolutions in BNNs, enhancing expressiveness and training stability with innovative convolution and residual methods.

Findings

Achieves state-of-the-art BNN performance on ImageNet with MobileNet V1.

Outperforms prior methods on CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet, and Oxford Flowers 102.

Improves accuracy by up to 9.3 percentage points across datasets.

Abstract

Recent advances in model compression have highlighted the potential of low-bit precision techniques, with Binary Neural Networks (BNNs) attracting attention for their extreme efficiency. However, extreme quantization in BNNs limits representational capacity and destabilizes training, posing significant challenges for lightweight architectures with depth-wise convolutions. To address this, we propose a 1.58-bit convolution to enhance expressiveness and a pre-BN residual connection to stabilize optimization by improving the Hessian condition number. These innovations enable, to the best of our knowledge, the first successful binarization of depth-wise convolutions in BNNs. Our method achieves 33M OPs on ImageNet with MobileNet V1, establishing a new state-of-the-art in BNNs by outperforming prior methods with comparable OPs. Moreover, it consistently outperforms existing methods across various datasets, including CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet, and Oxford Flowers 102, with accuracy improvements of up to 9.3 percentage points.