Training DNNs with Hybrid Block Floating Point

TL;DR

This paper introduces HBFP, a hybrid BFP-FP approach for DNN training that combines the accuracy of floating point with the hardware efficiency of fixed point, achieving up to 8.5x higher throughput.

Contribution

The paper proposes HBFP, a novel hybrid BFP-FP method that enables efficient DNN training with high accuracy and improved hardware performance.

Findings

HBFP matches floating point accuracy across various models.

HBFP achieves up to 8.5x higher throughput.

HBFP leverages BFP for dot products and floating point for other operations.

Abstract

The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5x higher throughput.