Theory-optimal Quantization Based on Flatness

TL;DR

This paper introduces a theory-driven quantization method for large language models that minimizes outlier effects, improving accuracy at low bit precision through a novel metric and optimized matrix transformations.

Contribution

It presents a new metric called Flatness, derives the theoretical optimal solution for quantization, and proposes BDQ, a framework that disperses outliers to enhance model compression.

Findings

BDQ achieves less than 1% accuracy drop on LLaMA-3-8B at W4A4 quantization.

BDQ reduces the performance gap by 39.1% on LLaMA-70B in challenging quantization settings.

Theoretical analysis links quantization error to outliers, guiding the design of effective transformations.

Abstract

Post-training quantization has emerged as a widely adopted technique for compressing and accelerating the inference of Large Language Models (LLMs). The primary challenges in LLMs quantization stem from activation outliers, which significantly degrade model performance especially at lower bit precision. While recent approaches attempt to mitigate outliers through linear transformations across feature dimensions, our analysis reveals that the transformed weights and activations still exhibit persistent outlier patterns with concentrated magnitude distributions. In this paper, we first model the mathematical relationship between quantization error and outliers, and then introduce a new metric Flatness to quantify the distribution of outliers. Based on this, we derive the theoretical optimal solution with respect to Flatness. Building on these insights, we propose Bidirectional Diagonal Quantization (BDQ), a novel post-training quantization framework that effectively disperses outlier patterns through optimized matrix transformations. BDQ strategically distributes outlier magnitudes across matrix dimensions via learned diagonal operations. Extensive experiments demonstrate that BDQ establishes a new quantization benchmark. It achieves less than 1\% accuracy drop in W4A4 quantization on the LLaMA-3-8B model. In the more challenging W2A4KV16 experiment, compared to state-of-the-art approaches, BDQ reduces the performance gap by 39.1\% on the DeepSeek-R1-Distill-LLaMA-70B model.