BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation

TL;DR

This paper introduces BiTAT, a novel quantization-aware training method that uses task-dependent transformations to effectively binarize neural networks, especially compact models, with minimal performance loss.

Contribution

The paper proposes a new orthonormal transformation approach that disentangles weights based on importance, improving extreme quantization performance for edge-device neural networks.

Findings

Reduces performance loss in 1-bit quantized MobileNets on ImageNet.

Preserves full-precision accuracy on CIFAR-100 with compact backbones.

Outperforms existing quantization baselines in various benchmarks.

Abstract

Neural network quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation, while preserving the performance of the original model. However, extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures (e.g., MobileNets) often used for edge-device deployments results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration even with extreme quantization by focusing on the inter-weight dependencies, between the weights within each layer and across consecutive layers. To minimize the quantization impact of each weight on others, we perform an orthonormal transformation of the weights at each layer by training an input-dependent correlation matrix and importance vector, such that each weight is disentangled from the others. Then, we quantize the weights based on their importance to minimize the loss of the information from the original weights/activations. We further perform progressive layer-wise quantization from the bottom layer to the top, so that quantization at each layer reflects the quantized distributions of weights and activations at previous layers. We validate the effectiveness of our method on various benchmark datasets against strong neural quantization baselines, demonstrating that it alleviates the performance degeneration on ImageNet and successfully preserves the full-precision model performance on CIFAR-100 with compact backbone networks.