Exceeding the Numerical and Performance Characteristics of IEEE-754 SGEMM with BFloat16 Tensor Cores on GPUs for Scientific Computing

TL;DR

This paper demonstrates that using BF16 Tensor Cores on modern GPUs can surpass native FP32 in performance and numerical accuracy for scientific matrix multiplication, leveraging hardware features like integrated scaling.

Contribution

It provides a detailed analysis and implementation of BF16 emulation on GPUs, showing it exceeds native FP32 in performance and numerical characteristics for scientific computing.

Findings

BF16 emulation on GPUs outperforms native FP32 in speed.

BF16-based matrix multiplication maintains or improves numerical accuracy.

The approach is library-ready and handles denormal numbers correctly.

Abstract

Largely due to their increased native capacity for numerical intensity and power efficiency, reduced-precision floating-point computing resources, primarily used in artificial intelligence (AI) applications, have expanded at a greater rate than their higher-precision relatives. This has led to various efforts focused upon leveraging plentiful reduced-precision hardware to mimic higher-precision mathematical calculations. This paper studies a specific use case, namely the use of bfloat16 (BF16) Tensor Cores found on modern GPUs in service of single precision (FP32) matrix multiply operations. Given that BF16 and FP32 share the same dynamic range, the option to accumulate BF16 operations into FP32 accumulators (at full-speed), and additional BF16 arithmetic characteristics specific to the Blackwell GPU architecture, such as integrated scaling hardware, such emulation is highly motivated. This paper examines the performance, efficiency, power, and numerical characteristics of FP32 matrix multiplication via BF16-based emulation and demonstrates how it exceeds numerical and performance characteristics of native FP32 for scientific applications. We also discuss a full library-ready implementation that correctly deals with denormals.