CTMQ: Cyclic Training of Convolutional Neural Networks with Multiple Quantization Steps

TL;DR

This paper introduces a cyclic training method for low-bit quantized CNNs that iteratively refines weights across multiple quantization steps, significantly improving accuracy on complex datasets like ImageNet.

Contribution

The proposed cyclic training approach with multiple quantization steps enhances low-bit CNN performance by leveraging iterative knowledge transfer from higher-precision models.

Findings

Improved Top-1 accuracy by 5.80% on ImageNet

Enhanced Top-5 accuracy by 6.85% on ImageNet

Effective in training binarized ResNet-18 models

Abstract

This paper proposes a training method having multiple cyclic training for achieving enhanced performance in low-bit quantized convolutional neural networks (CNNs). Quantization is a popular method for obtaining lightweight CNNs, where the initialization with a pretrained model is widely used to overcome degraded performance in low-resolution quantization. However, large quantization errors between real values and their low-bit quantized ones cause difficulties in achieving acceptable performance for complex networks and large datasets. The proposed training method softly delivers the knowledge of pretrained models to low-bit quantized models in multiple quantization steps. In each quantization step, the trained weights of a model are used to initialize the weights of the next model with the quantization bit depth reduced by one. With small change of the quantization bit depth, the performance gap can be bridged, thus providing better weight initialization. In cyclic training, after training a low-bit quantized model, its trained weights are used in the initialization of its accurate model to be trained. By using better training ability of the accurate model in an iterative manner, the proposed method can produce enhanced trained weights for the low-bit quantized model in each cycle. Notably, the training method can advance Top-1 and Top-5 accuracies of the binarized ResNet-18 on the ImageNet dataset by 5.80% and 6.85%, respectively.