FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding

TL;DR

FAST introduces a variable precision block floating point system with stochastic rounding for DNN training, significantly accelerating training time and reducing hardware resources while maintaining accuracy.

Contribution

The paper presents a novel FAST system that supports variable precision BFP for DNN training, enabling incremental precision increases and faster training.

Findings

Achieves 2-6× speedup in training time.

Maintains comparable validation accuracy to prior methods.

Reduces hardware resource usage during training.

Abstract

Block Floating Point (BFP) can efficiently support quantization for Deep Neural Network (DNN) training by providing a wide dynamic range via a shared exponent across a group of values. In this paper, we propose a Fast First, Accurate Second Training (FAST) system for DNNs, where the weights, activations, and gradients are represented in BFP. FAST supports matrix multiplication with variable precision BFP input operands, enabling incremental increases in DNN precision throughout training. By increasing the BFP precision across both training iterations and DNN layers, FAST can greatly shorten the training time while reducing overall hardware resource usage. Our FAST Multipler-Accumulator (fMAC) supports dot product computations under multiple BFP precisions. We validate our FAST system on multiple DNNs with different datasets, demonstrating a 2-6$\times$ speedup in training on a single-chip platform over prior work based on \textbf{mixed-precision or block} floating point number systems while achieving similar performance in validation accuracy.