INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

TL;DR

This study systematically compares floating-point and integer quantization formats for AI hardware, revealing that fine-grained integer formats like MXINT8 often outperform FP formats in accuracy and efficiency, challenging current industry trends.

Contribution

It provides a comprehensive analysis of FP versus INT quantization trade-offs, introduces a symmetric clipping method for INT training, and offers guidance for hardware-software co-design in AI accelerators.

Findings

MXINT8 outperforms FP in accuracy and efficiency at 8-bit fine-grained quantization.

FP formats have an advantage at 4-bit, but NVINT4 can surpass NVFP4 with outlier mitigation.

Symmetric clipping enables nearly lossless INT8 training, improving practical deployment.

Abstract

Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats to handle the pervasive activation outliers in Large Language Models (LLMs). Despite this industry trend, a unified comparison of FP and integer (INT) quantization across varying granularities has been missing, leaving algorithm and hardware co-design without clear guidance. This paper fills that gap by systematically investigating the trade-offs between FP and INT formats. We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced. Our comprehensive comparison demonstrates that for popular 8-bit fine-grained formats (e.g., MX with block size 32), MXINT8 is superior to its FP counterpart in both algorithmic accuracy and hardware efficiency. However, for 4-bit formats, FP (e.g., MXFP4, NVFP4) often holds an accuracy advantage , though we show that NVINT4 can surpass NVFP4 when outlier-mitigation techniques like Hadamard rotation are applied. We also introduce a symmetric clipping method that resolves gradient bias in fine-grained low-bit INT training, enabling nearly lossless performance for MXINT8 training. These findings challenge the current hardware trajectory, demonstrating that a one-size-fits-all FP approach is suboptimal and advocating that fine-grained INT formats, particularly MXINT8, offer a better balance of accuracy, power, and efficiency for future AI accelerators.