Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training

TL;DR

This paper introduces Accuracy Booster, a 4-bit fixed-point arithmetic method for DNN training that maintains accuracy while significantly increasing arithmetic density, outperforming existing formats.

Contribution

It proposes a novel 4-bit fixed-point format called Accuracy Booster, optimizing DNN training efficiency by leveraging single-level scaled HBFP and mixed-mantissa techniques.

Findings

Achieves over 2.3x higher arithmetic density than SOTA formats.

Maintains state-of-the-art accuracy with 4-bit training.

Uses 6-bit mantissas only in specific training phases.

Abstract

The unprecedented demand for computing resources to train DNN models has led to a search for minimal numerical encoding. Recent state-of-the-art (SOTA) proposals advocate for multi-level scaled narrow bitwidth numerical formats. In this paper, we show that single-level scaling is sufficient to maintain training accuracy while maximizing arithmetic density. We identify a previously proposed single-level scaled format for 8-bit training, Hybrid Block Floating Point (HBFP), as the optimal candidate to minimize. We perform a full-scale exploration of the HBFP design space using mathematical tools to study the interplay among various parameters and identify opportunities for even smaller encodings across layers and epochs. Based on our findings, we propose Accuracy Booster, a mixed-mantissa HBFP technique that uses 4-bit mantissas for over 99% of all arithmetic operations in training and 6-bit mantissas only in the last epoch and first/last layers. We show Accuracy Booster enables increasing arithmetic density over all other SOTA formats by at least 2.3x while achieving state-of-the-art accuracies in 4-bit training.