With Shared Microexponents, A Little Shifting Goes a Long Way

TL;DR

This paper presents Block Data Representations (BDR) and introduces shared microexponent (MX) formats that significantly improve quantization efficiency and performance in deep learning models, outperforming existing standards.

Contribution

It proposes a novel shared microexponent quantization format within the BDR framework, enabling better performance and flexibility for narrow-precision deep learning applications.

Findings

MX outperforms state-of-the-art quantization methods

Effective on large-scale generative models

Improves efficiency in recommendation systems

Abstract

This paper introduces Block Data Representations (BDR), a framework for exploring and evaluating a wide spectrum of narrow-precision formats for deep learning. It enables comparison of popular quantization standards, and through BDR, new formats based on shared microexponents (MX) are identified, which outperform other state-of-the-art quantization approaches, including narrow-precision floating-point and block floating-point. MX utilizes multiple levels of quantization scaling with ultra-fine scaling factors based on shared microexponents in the hardware. The effectiveness of MX is demonstrated on real-world models including large-scale generative pretraining and inferencing, and production-scale recommendation systems.