Elastic-Link for Binarized Neural Network

TL;DR

This paper introduces the Elastic-Link module to improve binarized neural networks, especially for 1x1 convolutions, significantly enhancing accuracy on ImageNet and enabling wider adoption of BNNs.

Contribution

The Elastic-Link module is a novel, easily integrated method that enriches information flow in BNNs, particularly improving performance on modern architectures with 1x1 convolutions.

Findings

Improves ResNet26 top-1 accuracy from 57.9% to 64.0%.

Achieves 56.4% top-1 accuracy with binarized MobileNet.

Sets new state-of-the-art of 71.9% top-1 accuracy with ReActNet.

Abstract

Recent work has shown that Binarized Neural Networks (BNNs) are able to greatly reduce computational costs and memory footprints, facilitating model deployment on resource-constrained devices. However, in comparison to their full-precision counterparts, BNNs suffer from severe accuracy degradation. Research aiming to reduce this accuracy gap has thus far largely focused on specific network architectures with few or no 1x1 convolutional layers, for which standard binarization methods do not work well. Because 1x1 convolutions are common in the design of modern architectures (e.g. GoogleNet, ResNet, DenseNet), it is crucial to develop a method to binarize them effectively for BNNs to be more widely adopted. In this work, we propose an "Elastic-Link" (EL) module to enrich information flow within a BNN by adaptively adding real-valued input features to the subsequent convolutional output features. The proposed EL module is easily implemented and can be used in conjunction with other methods for BNNs. We demonstrate that adding EL to BNNs produces a significant improvement on the challenging large-scale ImageNet dataset. For example, we raise the top-1 accuracy of binarized ResNet26 from 57.9% to 64.0%. EL also aids convergence in the training of binarized MobileNet, for which a top-1 accuracy of 56.4% is achieved. Finally, with the integration of ReActNet, it yields a new state-of-the-art result of 71.9% top-1 accuracy.