Quantization Error as a Metric for Dynamic Precision Scaling in Neural Net Training

TL;DR

This paper introduces a dynamic precision scaling method for neural network training that uses quantization error as a metric, enabling reduced bit-widths while maintaining high accuracy.

Contribution

It proposes a novel DPS scheme utilizing stochastic fixed-point rounding and quantization-error based scaling to adapt precision during training.

Findings

Achieved 98.8% test accuracy on MNIST with ~16 bits for weights and 14 bits for activations.

Reduced computational cost by lowering bit-widths without sacrificing accuracy.

Demonstrated effectiveness of quantization-error as a dynamic scaling metric.

Abstract

Recent work has explored reduced numerical precision for parameters, activations, and gradients during neural network training as a way to reduce the computational cost of training (Na & Mukhopadhyay, 2016) (Courbariaux et al., 2014). We present a novel dynamic precision scaling (DPS) scheme. Using stochastic fixed-point rounding, a quantization-error based scaling scheme, and dynamic bit-widths during training, we achieve 98.8% test accuracy on the MNIST dataset using an average bit-width of just 16 bits for weights and 14 bits for activations, compared to the standard 32-bit floating point values used in deep learning frameworks.