GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networks

TL;DR

This paper introduces GDNSQ, a novel method for low-bit neural network quantization that dynamically learns noise scale and bit-width, achieving high accuracy even at extreme low-bit settings.

Contribution

It proposes a differentiable quantization approach with learnable parameters and a penalty mechanism, enabling effective training of ultra-low-bit neural networks.

Findings

Achieves competitive accuracy at W1A1 quantization.

Maintains efficiency of Straight-Through Estimator (STE).

Effectively models capacity dynamics during quantization.

Abstract

Quantized neural networks can be viewed as a chain of noisy channels, where rounding in each layer reduces capacity as bit-width shrinks; the floating-point (FP) checkpoint sets the maximum input rate. We track capacity dynamics as the average bit-width decreases and identify resulting quantization bottlenecks by casting fine-tuning as a smooth, constrained optimization problem. Our approach employs a fully differentiable Straight-Through Estimator (STE) with learnable bit-width, noise scale and clamp bounds, and enforces a target bit-width via an exterior-point penalty; mild metric smoothing (via distillation) stabilizes training. Despite its simplicity, the method attains competitive accuracy down to the extreme W1A1 setting while retaining the efficiency of STE.