Widening and Squeezing: Towards Accurate and Efficient QNNs

TL;DR

This paper proposes a novel approach to improve quantization neural networks by projecting features into high-dimensional spaces and eliminating redundancies, achieving near full-precision performance with fewer parameters.

Contribution

It introduces a method to enhance QNN representation capability through high-dimensional feature projection and redundancy elimination, resulting in efficient models with comparable accuracy.

Findings

Achieves 29.9% top-1 error with binary ResNet-18 on ImageNet

Reduces parameters and calculations significantly

Maintains performance close to full-precision models

Abstract

Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters. Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques. However, we find the representation capability of quantization features is far weaker than full-precision features by experiments. We address this problem by projecting features in original full-precision networks to high-dimensional quantization features. Simultaneously, redundant quantization features will be eliminated in order to avoid unrestricted growth of dimensions for some datasets. Then, a compact quantization neural network but with sufficient representation ability will be established. Experimental results on benchmark datasets demonstrate that the proposed method is able to establish QNNs with much less parameters and calculations but almost the same performance as that of full-precision baseline models, e.g. $29.9\%$ top-1 error of binary ResNet-18 on the ImageNet ILSVRC 2012 dataset.