ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

TL;DR

ZipCache introduces a novel KV cache quantization method for large language models that significantly reduces memory and latency while maintaining high accuracy, by accurately identifying salient tokens and employing efficient quantization schemes.

Contribution

The paper proposes ZipCache, a new quantization approach that improves KV cache compression accuracy and efficiency through channel-separable quantization and normalized attention score-based saliency detection.

Findings

Achieves 4.98x KV cache compression with only 0.38% accuracy loss on GSM8k.

Reduces prefill latency by 37.3%, decoding latency by 56.9%, and GPU memory by 19.8%.

Outperforms previous KV cache compression methods in speed and accuracy retention.

Abstract

KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance. However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for LLMs. First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization. To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention. Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by $4.98\times$, with only a $0.38\%$ drop in accuracy. In terms of efficiency, ZipCache also showcases a $37.3\%$ reduction in prefill-phase latency, a $56.9\%$ reduction in decoding-phase latency, and a $19.8\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of $4096$.