HiFloat4 Format for Language Model Inference

TL;DR

This paper presents HiFloat4, a new 4-bit data format for language model inference that improves accuracy and efficiency by using shared scaling metadata and enabling fixed-point matrix multiplications.

Contribution

Introduction of HiFloat4, a block floating-point format with shared scaling for 4-bit elements, enhancing model accuracy and hardware efficiency in language model inference.

Findings

HiF4 outperforms NVFP4 in accuracy across multiple models.

HiF4 reduces hardware area and power consumption.

Effective for diverse language models and tasks.

Abstract

This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning. Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value. The metadata specifies a three-level scaling hierarchy, capturing inter- and intra-group dynamic range while improving the utilization of the representational space. In addition, the large 64-element group size enables matrix multiplications to be executed in a highly fixed-point manner, significantly reducing hardware area and power consumption. To evaluate the proposed format, we conducted inference experiments on several language models, including LLaMA, Qwen, Mistral, DeepSeek-V3.1 and LongCat. Results show that HiF4 achieves higher average accuracy than the state-of-the-art NVFP4 format across multiple models and diverse downstream tasks.