GPTVQ: The Blessing of Dimensionality for LLM Quantization

TL;DR

GPTVQ introduces a novel high-dimensional vector quantization method that significantly improves the size-accuracy trade-off for large language models, enabling efficient post-training quantization with state-of-the-art results.

Contribution

The paper presents GPTVQ, a fast, scalable post-training quantization method leveraging high-dimensional vector quantization and Hessian-based updates for large language models.

Findings

Achieves new state-of-the-art size-accuracy trade-offs on Llama-v2 and Mistral models.

Efficiently processes large models within 3-11 hours on H100 hardware.

VQ decoding on mobile CPUs offers latency improvements over 4-bit integer formats.

Abstract

In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose the GPTVQ method, a new fast method for post-training vector quantization (VQ) that scales well to Large Language Models (LLMs). Our method interleaves quantization of one or more columns with updates to the remaining unquantized weights, using information from the Hessian of the per-layer output reconstruction MSE. Quantization codebooks are initialized using an efficient data-aware version of the EM algorithm. The codebooks are then updated, and further compressed by using integer quantization and SVD-based compression. GPTVQ establishes a new state-of-the art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2 and Mistral. Furthermore, our method is efficient: on a single H100 it takes between 3 and 11 hours to process a Llamav2-70B model, depending on quantization setting. Lastly, with on-device timings for VQ decompression on a mobile CPU we show that VQ leads to improved latency compared to using a 4-bit integer format.