Accurate and Compact Convolutional Neural Networks with Trained Binarization

TL;DR

This paper introduces an improved training method for binary CNNs that uses trainable scaling factors and specialized algorithms to significantly enhance accuracy while maintaining compactness, enabling better deployment on resource-limited devices.

Contribution

The paper presents a novel training approach for binary CNNs incorporating trainable scaling factors and a specialized training algorithm, leading to higher accuracy than previous binary models.

Findings

Achieves 92.3% accuracy on CIFAR-10 with VGG-Small

Obtains 46.1% top-1 accuracy on ImageNet with AlexNet

Surpasses previous binary CNN methods in accuracy

Abstract

Although convolutional neural networks (CNNs) are now widely used in various computer vision applications, its huge resource demanding on parameter storage and computation makes the deployment on mobile and embedded devices difficult. Recently, binary convolutional neural networks are explored to help alleviate this issue by quantizing both weights and activations with only 1 single bit. However, there may exist a noticeable accuracy degradation when compared with full-precision models. In this paper, we propose an improved training approach towards compact binary CNNs with higher accuracy. Trainable scaling factors for both weights and activations are introduced to increase the value range. These scaling factors will be trained jointly with other parameters via backpropagation. Besides, a specific training algorithm is developed including tight approximation for derivative of discontinuous binarization function and $L_2$ regularization acting on weight scaling factors. With these improvements, the binary CNN achieves 92.3% accuracy on CIFAR-10 with VGG-Small network. On ImageNet, our method also obtains 46.1% top-1 accuracy with AlexNet and 54.2% with Resnet-18 surpassing previous works.