Trained Ternary Quantization

TL;DR

This paper introduces Trained Ternary Quantization (TTQ), a method to reduce neural network weights to ternary values, significantly decreasing model size with minimal accuracy loss and even improving performance on certain datasets.

Contribution

The paper presents a novel trained quantization approach that learns ternary weights and assignment, enabling efficient, smaller models trained from scratch with minimal accuracy degradation.

Findings

TTQ models are nearly 16x smaller than full-precision models.

TTQ outperforms full-precision models on CIFAR-10 with ResNet architectures.

TTQ achieves higher accuracy than previous ternary models on ImageNet.

Abstract

Deep neural networks are widely used in machine learning applications. However, the deployment of large neural networks models can be difficult to deploy on mobile devices with limited power budgets. To solve this problem, we propose Trained Ternary Quantization (TTQ), a method that can reduce the precision of weights in neural networks to ternary values. This method has very little accuracy degradation and can even improve the accuracy of some models (32, 44, 56-layer ResNet) on CIFAR-10 and AlexNet on ImageNet. And our AlexNet model is trained from scratch, which means it's as easy as to train normal full precision model. We highlight our trained quantization method that can learn both ternary values and ternary assignment. During inference, only ternary values (2-bit weights) and scaling factors are needed, therefore our models are nearly 16x smaller than full-precision models. Our ternary models can also be viewed as sparse binary weight networks, which can potentially be accelerated with custom circuit. Experiments on CIFAR-10 show that the ternary models obtained by trained quantization method outperform full-precision models of ResNet-32,44,56 by 0.04%, 0.16%, 0.36%, respectively. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.