QTIP: Quantization with Trellises and Incoherence Processing

TL;DR

QTIP leverages trellis coded quantization to enable ultra-high-dimensional weight quantization in LLMs, significantly improving inference efficiency and quantization quality over traditional vector quantization methods.

Contribution

The paper introduces QTIP, a novel quantization method using trellis coded quantization for high-dimensional weight compression in LLMs, overcoming limitations of vector quantization.

Findings

QTIP achieves state-of-the-art quantization quality.

QTIP improves inference speed in LLMs.

QTIP enables ultra-high-dimensional quantization.

Abstract

Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes. Since LLM inference is usually memory-bound, PTQ methods can improve inference throughput. Recent state-of-the-art PTQ approaches use vector quantization (VQ) to quantize multiple weights at once, which improves information utilization through better shaping. However, VQ requires a codebook with size exponential in the dimension. This limits current VQ-based PTQ works to low VQ dimensions ($\le 8$) that in turn limit quantization quality. Here, we introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization. TCQ uses a stateful decoder that separates the codebook size from the bitrate and effective dimension. QTIP introduces a spectrum of lookup-only to computed lookup-free trellis codes designed for a hardware-efficient "bitshift" trellis structure; these codes achieve state-of-the-art results in both quantization quality and inference speed.