Ternary Neural Networks with Fine-Grained Quantization

TL;DR

This paper introduces a fine-grained quantization method for ternarizing neural network weights and constraining activations to 8 and 4 bits, achieving high accuracy with significant reduction in multiplications, enabling efficient low-bit inference.

Contribution

The paper presents a novel FGQ approach for ternarizing pre-trained models with minimal accuracy loss and provides an improved theoretical formulation for better quantization quality.

Findings

Achieves within 3.7-4.2% of full precision accuracy on ResNet-50/101.

Eliminates up to 75% of multiplications without retraining.

Enables a full 8/4-bit inference pipeline with up to 15x performance improvement.

Abstract

We propose a novel fine-grained quantization (FGQ) method to ternarize pre-trained full precision models, while also constraining activations to 8 and 4-bits. Using this method, we demonstrate a minimal loss in classification accuracy on state-of-the-art topologies without additional training. We provide an improved theoretical formulation that forms the basis for a higher quality solution using FGQ. Our method involves ternarizing the original weight tensor in groups of $N$ weights. Using $N=4$, we achieve Top-1 accuracy within $3.7\%$ and $4.2\%$ of the baseline full precision result for Resnet-101 and Resnet-50 respectively, while eliminating $75\%$ of all multiplications. These results enable a full 8/4-bit inference pipeline, with best-reported accuracy using ternary weights on ImageNet dataset, with a potential of $9\times$ improvement in performance. Also, for smaller networks like AlexNet, FGQ achieves state-of-the-art results. We further study the impact of group size on both performance and accuracy. With a group size of $N=64$, we eliminate $\approx99\%$ of the multiplications; however, this introduces a noticeable drop in accuracy, which necessitates fine tuning the parameters at lower precision. We address this by fine-tuning Resnet-50 with 8-bit activations and ternary weights at $N=64$, improving the Top-1 accuracy to within $4\%$ of the full precision result with $<30\%$ additional training overhead. Our final quantized model can run on a full 8-bit compute pipeline using 2-bit weights and has the potential of up to $15\times$ improvement in performance compared to baseline full-precision models.