R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

TL;DR

R-KV is a novel redundancy-aware KV cache compression method that significantly reduces memory usage and improves inference throughput in reasoning models by preserving performance with much smaller caches.

Contribution

The paper introduces R-KV, a new approach that effectively compresses KV caches by removing redundancy, outperforming existing methods in reasoning model inference.

Findings

Achieves nearly 100% of full KV cache performance with only 10% cache size.

Outperforms existing KV cache baselines, reaching 105% performance with 16% cache.

Reduces memory usage by 90% and increases throughput by 6.6 times.

Abstract

Reasoning models have demonstrated impressive performance in self-reflection and chain-of-thought reasoning. However, they often produce excessively long outputs, leading to prohibitively large key-value (KV) caches during inference. While chain-of-thought inference significantly improves performance on complex reasoning tasks, it can also lead to reasoning failures when deployed with existing KV cache compression approaches. To address this, we propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a novel method specifically targeting redundant tokens in reasoning models. Our method preserves nearly 100% of the full KV cache performance using only 10% of the KV cache, substantially outperforming existing KV cache baselines, which reach only 60% of the performance. Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache. This KV-cache reduction also leads to a 90% memory saving and a 6.6X throughput over standard chain-of-thought reasoning inference. Experimental results show that R-KV consistently outperforms existing KV cache compression baselines across two mathematical reasoning datasets.