QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead

TL;DR

QJL introduces a zero-overhead 1-bit quantization method for KV cache in LLMs, significantly reducing memory usage and accelerating inference without accuracy loss by combining Johnson-Lindenstrauss transform with sign-bit quantization.

Contribution

The paper proposes QJL, a novel quantization technique that eliminates memory overhead by removing the need for storing quantization constants, enabling efficient 3-bit KV cache quantization.

Findings

Over fivefold reduction in KV cache memory usage.

Maintains accuracy while reducing memory and increasing speed.

Effective across various LLMs and NLP tasks.

Abstract

Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block. Depending on the block size, this overhead can add 1 or 2 bits per quantized number. We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants. We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion. We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation. When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime. Codes are available at \url{https://github.com/amirzandieh/QJL}.