ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

TL;DR

ChunkKV introduces a semantic-aware KV cache compression method for long-context LLM inference, preserving linguistic integrity and improving efficiency, outperforming existing methods in accuracy and throughput.

Contribution

It redefines KV cache compression by using semantic chunks, maintaining context and meaning, and introduces a layer-wise index reuse technique to reduce computational overhead.

Findings

Improves throughput by 26.5% with index reuse.

Outperforms state-of-the-art methods by up to 8.7% in precision.

Maintains high compression ratios while preserving semantic integrity.

Abstract

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.