Lossless KV Cache Compression to 2%

TL;DR

This paper presents CLLA, a novel architecture that compresses large language model KV caches to under 2% of their original size without losing performance, enabling more efficient inference.

Contribution

Introduction of CLLA, a comprehensive framework combining attention reduction, layer sharing, and quantization for near-lossless KV cache compression in language models.

Findings

Achieves lossless performance on most tasks.

Reduces KV cache size to less than 2%.

Enhances inference efficiency significantly.

Abstract

Large language models have revolutionized data processing in numerous domains, with their ability to handle extended context reasoning receiving notable recognition. To speed up inference, maintaining a key-value (KV) cache memory is essential. Nonetheless, the growing demands for KV cache memory create significant hurdles for efficient implementation. This work introduces a novel architecture, Cross-Layer Latent Attention (CLLA), aimed at compressing the KV cache to less than 2% of its original size while maintaining comparable performance levels. CLLA integrates multiple aspects of KV cache compression, including attention head/dimension reduction, layer sharing, and quantization techniques, into a cohesive framework. Our extensive experiments demonstrate that CLLA achieves lossless performance on most tasks while utilizing minimal KV cache, marking a significant advancement in practical KV cache compression.