SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks

TL;DR

This paper introduces SQWA, a novel stochastic weight averaging method for low-precision deep neural networks, which improves their generalization by averaging multiple quantized models during training.

Contribution

The paper proposes a new optimization approach, SQWA, that enhances low-precision DNNs' generalization by combining model averaging with quantization and visualization techniques.

Findings

Achieved state-of-the-art results for 2-bit QDNNs on CIFAR-100 and ImageNet.

Quantized DNNs with SQWA are located near flat minima, indicating better generalization.

Performance exceeds previous non-uniform quantization methods despite using uniform quantization.

Abstract

Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima. We present a new quantized neural network optimization approach, stochastic quantized weight averaging (SQWA), to design low-precision DNNs with good generalization capability using model averaging. The proposed approach includes (1) floating-point model training, (2) direct quantization of weights, (3) capturing multiple low-precision models during retraining with cyclical learning rates, (4) averaging the captured models, and (5) re-quantizing the averaged model and fine-tuning it with low-learning rates. Additionally, we present a loss-visualization technique on the quantized weight domain to clearly elucidate the behavior of the proposed method. Visualization results indicate that a quantized DNN (QDNN) optimized with the proposed approach is located near the center of the flat minimum in the loss surface. With SQWA training, we achieved state-of-the-art results for 2-bit QDNNs on CIFAR-100 and ImageNet datasets. Although we only employed a uniform quantization scheme for the sake of implementation in VLSI or low-precision neural processing units, the performance achieved exceeded those of previous studies employing non-uniform quantization.