Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies

TL;DR

This paper introduces BCDNet, a binary neural network that models contextual dependencies using binary MLP blocks and dynamic embeddings, significantly improving accuracy on ImageNet-1K over previous binary models.

Contribution

The paper proposes binary MLP blocks and contextual dynamic embeddings to enhance BNNs' ability to model dependencies, leading to state-of-the-art performance.

Findings

BCDNet achieves 72.3% Top-1 accuracy on ImageNet-1K.

Outperforms ReActNet-A by 2.9% Top-1 accuracy.

Binary MLPs effectively model both local and long-range dependencies.

Abstract

Existing Binary Neural Networks (BNNs) mainly operate on local convolutions with binarization function. However, such simple bit operations lack the ability of modeling contextual dependencies, which is critical for learning discriminative deep representations in vision models. In this work, we tackle this issue by presenting new designs of binary neural modules, which enables BNNs to learn effective contextual dependencies. First, we propose a binary multi-layer perceptron (MLP) block as an alternative to binary convolution blocks to directly model contextual dependencies. Both short-range and long-range feature dependencies are modeled by binary MLPs, where the former provides local inductive bias and the latter breaks limited receptive field in binary convolutions. Second, to improve the robustness of binary models with contextual dependencies, we compute the contextual dynamic embeddings to determine the binarization thresholds in general binary convolutional blocks. Armed with our binary MLP blocks and improved binary convolution, we build the BNNs with explicit Contextual Dependency modeling, termed as BCDNet. On the standard ImageNet-1K classification benchmark, the BCDNet achieves 72.3% Top-1 accuracy and outperforms leading binary methods by a large margin. In particular, the proposed BCDNet exceeds the state-of-the-art ReActNet-A by 2.9% Top-1 accuracy with similar operations. Our code is available at https://github.com/Sense-GVT/BCDN