Block-Wise Dynamic-Precision Neural Network Training Acceleration via Online Quantization Sensitivity Analytics

TL;DR

DYNASTY is a novel block-wise dynamic-precision training framework that uses online sensitivity analytics to accelerate neural network training and reduce energy consumption without sacrificing accuracy.

Contribution

It introduces a fast online analytics method for data sensitivity and an adaptive bit-width map generator for stable low-bit quantized training.

Findings

Up to 5.1x training speedup on CIFAR-100 and ImageNet.

Up to 4.7x energy reduction with no accuracy loss.

Effective block-wise dynamic-precision quantization.

Abstract

Data quantization is an effective method to accelerate neural network training and reduce power consumption. However, it is challenging to perform low-bit quantized training: the conventional equal-precision quantization will lead to either high accuracy loss or limited bit-width reduction, while existing mixed-precision methods offer high compression potential but failed to perform accurate and efficient bit-width assignment. In this work, we propose DYNASTY, a block-wise dynamic-precision neural network training framework. DYNASTY provides accurate data sensitivity information through fast online analytics, and maintains stable training convergence with an adaptive bit-width map generator. Network training experiments on CIFAR-100 and ImageNet dataset are carried out, and compared to 8-bit quantization baseline, DYNASTY brings up to $5.1\times$ speedup and $4.7\times$ energy consumption reduction with no accuracy drop and negligible hardware overhead.