SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

TL;DR

The paper introduces SKVQ, a novel sliding-window KV cache quantization method that significantly reduces memory usage in large language models while preserving accuracy, enabling longer context processing and faster decoding.

Contribution

SKVQ is the first approach to effectively quantize KV caches to extremely low bitwidths with high compression and minimal accuracy loss.

Findings

Achieves 2-bit keys and 1.5-bit values quantization with minimal accuracy loss.

Enables processing of up to 1 million tokens on an 80GB GPU.

Provides up to 7x faster decoding for large language models.

Abstract

Large language models (LLMs) can now handle longer sequences of tokens, enabling complex tasks like book understanding and generating lengthy novels. However, the key-value (KV) cache required for LLMs consumes substantial memory as context length increasing, becoming the bottleneck for deployment. In this paper, we present a strategy called SKVQ, which stands for sliding-window KV cache quantization, to address the issue of extremely low bitwidth KV cache quantization. To achieve this, SKVQ rearranges the channels of the KV cache in order to improve the similarity of channels in quantization groups, and applies clipped dynamic quantization at the group level. Additionally, SKVQ ensures that the most recent window tokens in the KV cache are preserved with high precision. This helps maintain the accuracy of a small but important portion of the KV cache.SKVQ achieves high compression ratios while maintaining accuracy. Our evaluation on LLMs demonstrates that SKVQ surpasses previous quantization approaches, allowing for quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal loss of accuracy. With SKVQ, it is possible to process context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7 times faster decoding.