QAQ: Quality Adaptive Quantization for LLM KV Cache

TL;DR

QAQ introduces a novel, adaptive quantization method for LLM KV caches that significantly compresses cache size with minimal performance loss, enabling longer-context applications and more efficient deployment.

Contribution

The paper presents QAQ, a new quantization scheme that separately optimizes key and value caches, incorporating outlier handling and attention-awareness for superior compression.

Findings

Achieves up to 10x compression ratio of KV cache size

Negligible impact on model performance

Enables longer-context LLM applications

Abstract

The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress the KV cache and improve model throughput. However, heuristics used by these strategies may wrongly evict essential KV cache, which can significantly degrade model performance. In this paper, we propose QAQ, a Quality Adaptive Quantization scheme for the KV cache. We theoretically demonstrate that key cache and value cache exhibit distinct sensitivities to quantization, leading to the formulation of separate quantization strategies for their non-uniform quantization. Through the integration of dedicated outlier handling, as well as an improved attention-aware approach, QAQ achieves up to 10x the compression ratio of the KV cache size with a neglectable impact on model performance. QAQ significantly reduces the practical hurdles of deploying LLMs, opening up new possibilities for longer-context applications. The code is available at github.com/ClubieDong/KVCacheQuantization.