Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models

TL;DR

This paper introduces Benford-Quant, a non-uniform quantizer based on Benford's Law, which improves weight quantization in large language models by allocating more resolution to small weights, leading to better perplexity scores.

Contribution

The paper presents a novel Benford-inspired quantization method that is data-free, simple to implement, and enhances the accuracy of LLMs in low-bit regimes compared to uniform quantization.

Findings

Benford-Quant improves perplexity on small language models by over 10% with 4-bit quantization.

Transformer layer weights closely follow Benford's Law, unlike normalization layers.

Benford-Quant remains competitive on larger models, with benefits explained by over-parameterization effects.

Abstract

The rapid growth of Large Language Models (LLMs) intensifies the need for effective compression, with weight quantization being the most widely adopted technique. Standard uniform quantizers assume that parameters are evenly distributed, an assumption at odds with the highly skewed distributions observed in practice. We propose Benford-Quant, a simple, data-free non-uniform quantizer inspired by Benford's Law, which predicts that leading digits follow a logarithmic distribution. Benford-Quant replaces the uniform grid with a log-spaced codebook, dedicating more resolution to the frequent small-magnitude weights. We provide both theoretical intuition and empirical evidence: (i) weights in transformer transformational layers adhere closely to Benford statistics, while normalization layers systematically deviate; (ii) on Small Language Models (SLMs), Benford-Quant consistently improves perplexity, reducing 4-bit perplexity on Gemma-270M by more than 10%; and (iii) on larger LLMs, it remains competitive, with differences explained by over-parameterization effects. Our results indicate that incorporating a Benford-inspired prior into quantization grids is a low-cost modification that yields accuracy gains in aggressive few-bit regimes. Although it is not able to surpass the state of the art in tasks such as perplexity and LAMBADA, the Benford-Quant approach can be hybridized with other quantization methods-such as SmoothQuant and Activation-Aware Quantization-without major pipeline modification, potentially improving their performance.