M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization

TL;DR

This paper introduces M2XFP, a metadata-augmented data format for low-bit quantization that improves accuracy and efficiency in large language model acceleration through a novel algorithm-hardware co-design.

Contribution

It proposes a new metadata-based quantization method with a hardware implementation that significantly reduces accuracy loss and enhances speed and energy efficiency.

Findings

Achieves 70.63% less accuracy loss compared to MXFP4.

Provides up to 1.91× speedup and 1.75× energy savings.

Demonstrates effectiveness on large language model benchmarks.

Abstract

Existing low-bit Microscaling (MX) formats, such as MXFP4, often suffer from substantial accuracy degradation due to the use of a shared scaling factor with the Power-of-Two format. In this work, we explore strategies that introduce minimal metadata to recover accuracy lost during quantization while maintaining high bit efficiency across a wide range of large language models. We propose a complete algorithm-hardware co-design based on flexible metadata, featuring an online quantization with simple encoding. To support the proposed method efficiently, we implement a lightweight hardware unit and integrate it into the accelerator. Evaluation results demonstrate that our method substantially narrows the accuracy gap, achieving on average a 70.63% reduction in accuracy loss compared to MXFP4 and a 37.30% reduction relative to the latest NVFP4 on LLM benchmarks. Furthermore, our design delivers up to 1.91$\times$ speedup and 1.75$\times$ energy savings over state-of-the-art accelerators. Our code is available at https://github.com/SJTU-ReArch-Group/M2XFP_ASPLOS26.