DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization

TL;DR

DuQuant++ introduces an outlier-aware rotation technique tailored for MXFP4 microscaling format, significantly improving quantization accuracy and efficiency for large language models.

Contribution

It adapts fine-grained rotation to MXFP4, reducing complexity and enhancing quantization performance by specifically targeting activation outliers.

Findings

Achieves state-of-the-art quantization performance on LLaMA-3 models.

Halves the online rotation cost compared to previous methods.

Effectively smooths weight distribution and handles outliers.

Abstract

The MXFP4 microscaling format, which partitions tensors into blocks of 32 elements sharing an E8M0 scaling factor, has emerged as a promising substrate for efficient LLM inference, backed by native hardware support on NVIDIA Blackwell Tensor Cores. However, activation outliers pose a unique challenge under this format: a single outlier inflates the shared block scale, compressing the effective dynamic range of the remaining elements and causing significant quantization error. Existing rotation-based remedies, including randomized Hadamard and learnable rotations, are data-agnostic and therefore unable to specifically target the channels where outliers concentrate. We propose DuQuant++, which adapts the outlier-aware fine-grained rotation of DuQuant to the MXFP4 format by aligning the rotation block size with the microscaling group size (B{=}32). Because each MXFP4 group possesses an independent scaling factor, the cross-block variance issue that necessitates dual rotations and a zigzag permutation in the original DuQuant becomes irrelevant, enabling DuQuant++ to replace the entire pipeline with a single outlier-aware rotation, which halves the online rotation cost while simultaneously smoothing the weight distribution. Extensive experiments on the LLaMA-3 family under MXFP4 W4A4 quantization show that DuQuant++ consistently achieves state-of-the-art performance. Our code is available at https://github.com/Hsu1023/DuQuant-v2.