Pyramid Vector Quantization for LLMs

TL;DR

This paper introduces Pyramid Vector Quantization (PVQ) for large language models, leveraging spherical geometry and Hessian information to achieve state-of-the-art compression with minimal accuracy loss.

Contribution

It develops a practical PVQ algorithm combined with scale and Hessian-based quantization, enabling efficient, high-quality model compression without explicit codebooks.

Findings

Quantized Llama-3 70B to 3.25 bits per weight with 98% accuracy

Achieved state-of-the-art trade-off between compression and performance

Extended PVQ to incorporate Hessian information for error minimization

Abstract

Recent works on compression of large language models (LLM) using quantization considered reparameterizing the architecture such that weights are distributed on the sphere. This demonstratively improves the ability to quantize by increasing the mathematical notion of coherence, resulting in fewer weight outliers without affecting the network output. In this work, we aim to further exploit this spherical geometry of the weights when performing quantization by considering Pyramid Vector Quantization (PVQ) for large language models. Arranging points evenly on the sphere is notoriously difficult, especially in high dimensions, and in case approximate solutions exists, representing points explicitly in a codebook is typically not feasible due to its additional memory cost. Instead, PVQ uses a fixed integer lattice on the sphere by projecting points onto the 1-sphere, which allows for efficient encoding and decoding without requiring an explicit codebook in memory. To obtain a practical algorithm, we propose to combine PVQ with scale quantization for which we derive theoretically optimal quantizations, under empirically verified assumptions. Further, we extend pyramid vector quantization to use Hessian information to minimize quantization error under expected feature activations, instead of only relying on weight magnitudes. Experimentally, we achieves state-of-the-art quantization performance with pareto-optimal trade-off between performance and bits per weight and bits per activation, compared to compared methods. On weight-only, we find that we can quantize a Llama-3 70B model to 3.25 bits per weight and retain 98\% accuracy on downstream tasks.