NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Semyon Savkin, Eitan Porat, Or Ordentlich, Yury Polyanskiy

TL;DR
NestQuant introduces a nested lattice-based post-training quantization method for large language models, significantly reducing model size and perplexity while maintaining high accuracy across various models and benchmarks.
Contribution
The paper presents a practical, low-complexity nested lattice quantization scheme for LLMs, outperforming existing methods in perplexity reduction and model efficiency.
Findings
Quantizes Llama-3-8B weights and activations to 4 bits with perplexity 6.6
Achieves over 55% reduction in perplexity gap compared to unquantized models
Consistently outperforms state-of-the-art quantization methods on larger models and benchmarks
Abstract
Post-training quantization (PTQ) has emerged as a critical technique for efficient deployment of large language models (LLMs). This work proposes NestQuant, a novel PTQ scheme for weights and activations that is based on self-similar nested lattices. Recent works have mathematically shown such quantizers to be information-theoretically optimal for low-precision matrix multiplication. We implement a practical low-complexity version of NestQuant based on Gosset lattice, making it a drop-in quantizer for any matrix multiplication step (e.g., in self-attention, MLP etc). For example, NestQuant quantizes weights, KV-cache, and activations of Llama-3-8B to 4 bits, achieving perplexity of 6.6 on wikitext2. This represents more than 55% reduction in perplexity gap with respect to unquantized model (perplexity of 6.14) compared to state-of-the-art Metas SpinQuant (perplexity 7.3), OstQuant (7.3)…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsDistributed and Parallel Computing Systems
