Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling

TL;DR

This paper introduces Four Over Six (4/6), an adaptive block scaling method for NVFP4 quantization that reduces error and improves performance in large language models.

Contribution

The paper proposes a novel adaptive scaling technique for NVFP4 quantization, enhancing accuracy and efficiency in large language model training and inference.

Findings

4/6 reduces quantization error compared to existing NVFP4 methods.

Training loss with 4/6 is closer to BF16 than previous NVFP4 recipes.

Efficient implementation of 4/6 yields performance gains on modern hardware.

Abstract

As large language models have grown larger, interest has grown in low-precision numerical formats such as NVFP4 as a way to improve speed and reduce memory usage. However, quantizing models to NVFP4 remains challenging as the lack of precision generally degrades model performance. In this work, we address this issue with Four Over Six (4/6), a modification to the block-scaled NVFP4 quantization algorithm that yields reduced quantization error. Unlike integer formats, floating point formats have non-uniform step sizes which create larger quantization error on larger values. 4/6 takes advantage of this by adaptively scaling some blocks to smaller FP4 values, making the distribution of representable values more uniform and reducing quantization error for near-maximal values. We show that 4/6 can be implemented efficiently on modern hardware accelerators, resulting in performance gains during both pre-training and inference with minimal computational overhead. In pre-training experiments with the Nemotron 3 Nano 30B-A3B model architecture, we find that 4/6 brings training loss closer to BF16 compared to models trained with current state-of-the-art NVFP4 training recipes. Our code is available at https://github.com/mit-han-lab/fouroversix.