PQCache: Product Quantization-based KVCache for Long Context LLM Inference

TL;DR

PQCache leverages product quantization to efficiently manage key-value caches in large language models, significantly reducing memory bottlenecks and latency while maintaining model quality during long-context inference.

Contribution

The paper introduces PQCache, a novel method applying product quantization to KVCache in LLMs, balancing memory efficiency and inference quality.

Findings

Achieves 4.60% score improvement on InfiniteBench

Reduces serving latency during inference

Maintains model quality with low overhead

Abstract

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques prevalent in the data management community, we consider the storage and retrieval of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, we use PQ codes and centroids to approximately identify important preceding tokens, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments demonstrate that PQCache achieves both effectiveness and efficiency, with 4.60% score improvement over existing methods on InfiniteBench and low system latency in both prefilling and decoding.