Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks

TL;DR

This paper introduces two innovative techniques for training low bit-width deep neural networks, enabling direct weight quantization with learnable levels and channel-aware activation quantization, resulting in state-of-the-art accuracy.

Contribution

It presents a novel method for directly updating quantized weights with learnable levels and a channel-aware activation quantization approach, improving low-bit neural network training.

Findings

Achieves state-of-the-art accuracy on CIFAR-100 and ImageNet.

Effective low bit-width training for AlexNet, ResNet, MobileNetV2.

Outperforms existing quantization methods in accuracy.

Abstract

This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. However, this approach would result in some mismatch: the gradient descent updates full-precision weights, but it does not update the quantized weights. To address this issue, we propose a novel method that enables {direct} updating of quantized weights {with learnable quantization levels} to minimize the cost function using gradient descent. Second, to obtain low bit-width activations, existing works consider all channels equally. However, the activation quantizers could be biased toward a few channels with high-variance. To address this issue, we propose a method to take into account the quantization errors of individual channels. With this approach, we can learn activation quantizers that minimize the quantization errors in the majority of channels. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on the image classification task, using AlexNet, ResNet and MobileNetV2 architectures on CIFAR-100 and ImageNet datasets.