ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration

TL;DR

ReCalKV introduces a novel post-training low-rank KV cache compression method for large language models, employing head reordering and offline calibration to maintain performance while significantly reducing memory usage.

Contribution

It presents ReCalKV, a new approach that separately optimizes Key and Value caches using head-wise reordering and offline calibration, improving compression effectiveness.

Findings

ReCalKV outperforms existing methods in high compression scenarios.

It achieves minimal performance loss at high compression ratios.

The approach is validated through extensive experiments.

Abstract

Large language models (LLMs) have demonstrated remarkable performance, but their long-context reasoning remains constrained by the excessive memory required for the Key-Value (KV) cache. This makes KV cache compression a critical step toward efficient long-context inference. Recent methods have explored low-rank techniques to reduce the hidden size of the KV cache. However, they neglect the distinct roles and varying importance of Keys and Values, leading to significant performance drops under high compression. To address this, we propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values. For Keys, we propose Head-wise Similarity aware Reordering (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation via grouped SVD. For Values, we propose Offline Value Calibration (OVC), which efficiently calibrates the value projection matrix using calibration data without training, ensuring an accurate representation of contextual information. Extensive experiments show that ReCalKV consistently outperforms existing low-rank compression methods, achieving high compression ratios with minimal performance loss. The code and models will be available at:https://github.com/XIANGLONGYAN/ReCalKV.