Training Deep Neural Networks with 8-bit Floating Point Numbers

TL;DR

This paper demonstrates the first successful training of deep neural networks using only 8-bit floating point numbers, maintaining accuracy while significantly improving hardware efficiency.

Contribution

It introduces novel techniques for 8-bit floating point training, enabling high-accuracy DNN training with reduced precision, which was previously considered challenging.

Findings

Achieved accurate training of DNNs with 8-bit floating point numbers.

Reduced addition precision from 32-bit to 16-bit without loss of accuracy.

Potential for 2-4x hardware throughput improvements.

Abstract

The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.