QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework

TL;DR

QuantNet introduces a fully differentiable meta-based quantizer that directly binarizes weights without STE, effectively reducing gradient mismatching and discretization errors, leading to improved accuracy in binarized neural networks.

Contribution

It presents a novel differentiable framework for network quantization that eliminates the need for STE and enhances performance of binarized models.

Findings

Significant accuracy improvements on CIFAR-100 and ImageNet.

Bridges the gap between binarized and full-precision models.

Reduces discretization errors in quantized networks.

Abstract

Despite the achievements of recent binarization methods on reducing the performance degradation of Binary Neural Networks (BNNs), gradient mismatching caused by the Straight-Through-Estimator (STE) still dominates quantized networks. This paper proposes a meta-based quantizer named QuantNet, which utilizes a differentiable sub-network to directly binarize the full-precision weights without resorting to STE and any learnable gradient estimators. Our method not only solves the problem of gradient mismatching, but also reduces the impact of discretization errors, caused by the binarizing operation in the deployment, on performance. Generally, the proposed algorithm is implemented within a fully differentiable framework, and is easily extended to the general network quantization with any bits. The quantitative experiments on CIFAR-100 and ImageNet demonstrate that QuantNet achieves the signifficant improvements comparing with previous binarization methods, and even bridges gaps of accuracies between binarized models and full-precision models.