Statistically-Lossless Quantization of Large Language Models

TL;DR

This paper introduces a statistically-lossless quantization method for large language models that balances fidelity and efficiency, achieving significant compression and speedups while maintaining accuracy.

Contribution

It formalizes notions of task-lossless and distribution-lossless compression, proposes the EAR metric, and develops SLQ, a novel asymmetric quantization technique with wide bitwidth search.

Findings

Task-lossless compression achieved below 4 bits per parameter.

Distribution-lossless compression achieved at 5-6 bits per parameter.

Inference speedups of 1.7 to 3.6 times over FP16.

Abstract

Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but typically do not accelerate inference. This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric (for example, EAR >= 0.99 indicates 99% agreement). Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization, making asymmetry necessary for distribution-lossless fidelity but not for task-level preservation. Using SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we achieve task-lossless compression at well below 4 bits per parameter (as low as 3.3 bits depending on the model), distribution-lossless compression at 5 to 6 bits per parameter on average, and inference speedups of 1.7 to 3.6x relative to FP16 with optimized kernels. Source code is available at https://github.com/IST-DASLab/SLQ.