Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients

TL;DR

This paper proposes a novel method for low-bit fixed-point gradient quantization in neural network training, focusing on minimizing quantization error for large gradients to improve training efficiency and accuracy.

Contribution

It introduces an analysis of gradient quantization error and an adaptive interval update algorithm to optimize quantization for large gradients during training.

Findings

Significant performance improvement across multiple tasks.

Effective quantization for various network architectures.

Enhanced training stability with low-bit gradients.

Abstract

Network quantization generally converts full-precision weights and/or activations into low-bit fixed-point values in order to accelerate an inference process. Recent approaches to network quantization further discretize the gradients into low-bit fixed-point values, enabling an efficient training. They typically set a quantization interval using a min-max range of the gradients or adjust the interval such that the quantization error for entire gradients is minimized. In this paper, we analyze the quantization error of gradients for the low-bit fixed-point training, and show that lowering the error for large-magnitude gradients boosts the quantization performance significantly. Based on this, we derive an upper bound of quantization error for the large gradients in terms of the quantization interval, and obtain an optimal condition for the interval minimizing the quantization error for large gradients. We also introduce an interval update algorithm that adjusts the quantization interval adaptively to maintain a small quantization error for large gradients. Experimental results demonstrate the effectiveness of our quantization method for various combinations of network architectures and bit-widths on various tasks, including image classification, object detection, and super-resolution.