NIPQ: Noise proxy-based Integrated Pseudo-Quantization

TL;DR

NIPQ introduces a unified pseudoquantization method that improves quantization-aware training stability and performance across vision and language models by avoiding STE instability.

Contribution

The paper proposes NIPQ, a novel noise proxy-based pseudoquantization framework that supports both activation and weight quantization with gradient-based updates, enhancing stability and accuracy.

Findings

NIPQ outperforms existing algorithms in vision tasks.

NIPQ achieves significant improvements in language model quantization.

NIPQ maintains stable convergence without STE instability.

Abstract

Straight-through estimator (STE), which enables the gradient flow over the non-differentiable function via approximation, has been favored in studies related to quantization-aware training (QAT). However, STE incurs unstable convergence during QAT, resulting in notable quality degradation in low precision. Recently, pseudoquantization training has been proposed as an alternative approach to updating the learnable parameters using the pseudo-quantization noise instead of STE. In this study, we propose a novel noise proxy-based integrated pseudoquantization (NIPQ) that enables unified support of pseudoquantization for both activation and weight by integrating the idea of truncation on the pseudo-quantization framework. NIPQ updates all of the quantization parameters (e.g., bit-width and truncation boundary) as well as the network parameters via gradient descent without STE instability. According to our extensive experiments, NIPQ outperforms existing quantization algorithms in various vision and language applications by a large margin.