PolarQuant: Quantizing KV Caches with Polar Transformation

TL;DR

PolarQuant is a novel quantization method for KV caches in large language models that uses polar transformation and random preconditioning to significantly reduce memory usage while maintaining high quality.

Contribution

It introduces a new polar transformation-based quantization technique that eliminates normalization overhead, enabling over 4.2x memory compression of KV caches in LLMs.

Findings

Achieves over 4.2x compression of KV caches.

Maintains state-of-the-art quality scores.

Eliminates normalization step in quantization.

Abstract

Large language models (LLMs) require significant memory to store Key-Value (KV) embeddings in their KV cache, especially when handling long-range contexts. Quantization of these KV embeddings is a common technique to reduce memory consumption. This work introduces PolarQuant, a novel quantization method employing random preconditioning and polar transformation. Our method transforms the KV embeddings into polar coordinates using an efficient recursive algorithm and then quantizes resulting angles. Our key insight is that, after random preconditioning, the angles in the polar representation exhibit a tightly bounded and highly concentrated distribution with an analytically computable form. This nice distribution eliminates the need for explicit normalization, a step required by traditional quantization methods which introduces significant memory overhead because quantization parameters (e.g., zero point and scale) must be stored in full precision per each data block. PolarQuant bypasses this normalization step, enabling substantial memory savings. The long-context evaluation demonstrates that PolarQuant compresses the KV cache by over x4.2 while achieving the best quality scores compared to the state-of-the-art methods.