One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache

TL;DR

DynaKV is a post-training, token-wise adaptive compression method for KV caches in large language models, significantly reducing memory usage while preserving performance by dynamically allocating compression rates based on token semantics.

Contribution

It introduces DynaKV, the first dynamic, token-wise adaptive compression framework for KV caches that improves fidelity at high compression ratios without retraining.

Findings

Retains only 6% of KV cache while maintaining 94% performance.

Outperforms existing compression techniques in memory reduction and quality.

Compatible with sequence-level pruning methods.

Abstract

Despite the remarkable progress of Large Language Models (LLMs), the escalating memory footprint of the Key-Value (KV) cache remains a critical bottleneck for efficient inference. While dimensionality reduction offers a promising compression avenue, existing approaches typically either necessitate prohibitively expensive pre-training from scratch or suffer from severe performance deterioration under high compression regimes. In this work, we propose DynaKV, a novel post-training framework for low-rank KV cache compression. To the best of our knowledge, DynaKV is the first method to dynamically allocate compression rates to individual tokens according to their semantic meaning, which allows it to achieve better fidelity at aggressive compression ratios. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art compression techniques, achieving significant memory reduction while maintaining competitive generation quality. Furthermore, our approach is orthogonal to sequence-level pruning methods. When integrated with SnapKV, DynaKV retains only 6% of the KV cache while maintaining 94% of the baseline performance on the LongBench benchmark.