The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm

TL;DR

This paper reveals that GPTQ, a popular LLM quantization method, is mathematically equivalent to Babai's nearest plane algorithm, providing a geometric interpretation and enabling improved quantization techniques.

Contribution

It establishes a theoretical link between GPTQ and Babai's algorithm, leading to new error bounds and enhanced quantization methods for large language models.

Findings

GPTQ is equivalent to Babai's nearest plane algorithm for linear layers.

The geometric interpretation allows for error bounds and improved quantization avoiding clipping.

The paper provides GPU inference kernels for the new quantization approach.

Abstract

Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.