Symmetry Regularization and Saturating Nonlinearity for Robust Quantization

TL;DR

This paper introduces symmetry regularization and saturating nonlinearity to improve neural network robustness against quantization errors, enabling consistent performance across different bit-widths and low-precision implementations.

Contribution

It proposes two novel methods, SymReg and SatNL, that enhance network robustness during training, applicable to existing PTQ and QAT algorithms, and supports these with extensive empirical validation.

Findings

Enhanced robustness of neural networks against quantization errors.

Single weight configuration maintains output quality across various quantization conditions.

Validated effectiveness on CIFAR and ImageNet datasets.

Abstract

Robust quantization improves the tolerance of networks for various implementations, allowing reliable output in different bit-widths or fragmented low-precision arithmetic. In this work, we perform extensive analyses to identify the sources of quantization error and present three insights to robustify a network against quantization: reduction of error propagation, range clamping for error minimization, and inherited robustness against quantization. Based on these insights, we propose two novel methods called symmetry regularization (SymReg) and saturating nonlinearity (SatNL). Applying the proposed methods during training can enhance the robustness of arbitrary neural networks against quantization on existing post-training quantization (PTQ) and quantization-aware training (QAT) algorithms and enables us to obtain a single weight flexible enough to maintain the output quality under various conditions. We conduct extensive studies on CIFAR and ImageNet datasets and validate the effectiveness of the proposed methods.