Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs

TL;DR

AdaPT introduces a dynamic fixed-point quantization method for DNN training that adaptively adjusts precision per layer, leading to faster training and inference with minimal accuracy loss.

Contribution

It proposes a novel adaptive precision training strategy that optimizes bit-widths per layer based on information theory, improving resource efficiency over fixed global precision methods.

Findings

Achieves 1.27x training speedup over float32 baseline.

Attains 0.98% higher accuracy on CIFAR datasets.

Realizes 2.33x inference speedup with reduced model size.

Abstract

Quantization is a technique for reducing deep neural networks (DNNs) training and inference times, which is crucial for training in resource constrained environments or applications where inference is time critical. State-of-the-art (SOTA) quantization approaches focus on post-training quantization, i.e., quantization of pre-trained DNNs for speeding up inference. While work on quantized training exists, most approaches require refinement in full precision (usually single precision) in the final training phase or enforce a global word length across the entire DNN. This leads to suboptimal assignments of bit-widths to layers and, consequently, suboptimal resource usage. In an attempt to overcome such limitations, we introduce AdaPT, a new fixed-point quantized sparsifying training strategy. AdaPT decides about precision switches between training epochs based on information theoretic conditions. The goal is to determine on a per-layer basis the lowest precision that causes no quantization-induced information loss while keeping the precision high enough such that future learning steps do not suffer from vanishing gradients. The benefits of the resulting fully quantized DNN are evaluated based on an analytical performance model which we develop. We illustrate that an average speedup of 1.27 compared to standard training in float32 with an average accuracy increase of 0.98% can be achieved for AlexNet/ResNet on CIFAR10/100 and we further demonstrate these AdaPT trained models achieve an average inference speedup of 2.33 with a model size reduction of 0.52.