eXmY: A Data Type and Technique for Arbitrary Bit Precision Quantization

TL;DR

eXmY introduces a flexible data type and technique for arbitrary bit precision quantization in ML models, enabling efficient compression, storage, and computation across various hardware platforms.

Contribution

It presents a novel data type supporting arbitrary bit widths and formats, along with encoding schemes and implementations for efficient quantization in ML workflows.

Findings

Supports 3 to 9 bit formats with perfect compression.

Achieves high performance using SIMD and vector instructions.

Deployed in production for nearly 2 years.

Abstract

eXmY is a novel data type for quantization of ML models. It supports both arbitrary bit widths and arbitrary integer and floating point formats. For example, it seamlessly supports 3, 5, 6, 7, 9 bit formats. For a specific bit width, say 7, it defines all possible formats e.g. e0m6, e1m5, e2m4, e3m3, e4m2, e5m1 and e6m0. For non-power of two bit widths e.g. 5, 6, 7, we created a novel encoding and decoding scheme which achieves perfect compression, byte addressability and is amenable to sharding and vector processing. We implemented libraries for emulation, encoding and decoding tensors and checkpoints in C++, TensorFlow, JAX and PAX. For optimal performance, the codecs use SIMD instructions on CPUs and vector instructions on TPUs and GPUs. eXmY is also a technique and exploits the statistical distribution of exponents in tensors. It can be used to quantize weights, static and dynamic activations, gradients, master weights and optimizer state. It can reduce memory (CPU DRAM and accelerator HBM), network and disk storage and transfers. It can increase multi tenancy and accelerate compute. eXmY has been deployed in production for almost 2 years.