KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging

TL;DR

KVSlimmer introduces a theoretically grounded, gradient-free algorithm for efficient asymmetric KV merging in large language models, significantly reducing memory and latency while improving performance.

Contribution

It provides a novel theoretical framework for KV asymmetry and develops an exact, gradient-free algorithm that outperforms existing methods in practical benchmarks.

Findings

Outperforms SOTA methods on multiple benchmarks.

Reduces memory costs by 29% and latency by 28%.

Improves Llama3.1-8B-Instruct LongBench score by 0.92.

Abstract

The growing computational and memory demands of the Key-Value (KV) cache significantly limit the ability of Large Language Models (LLMs). While KV merging has emerged as a promising solution, existing methods that rely on empirical observations of KV asymmetry and gradient-based Hessian approximations lack a theoretical foundation and incur suboptimal compression and inference overhead. To bridge these gaps, we establish a theoretical framework that characterizes this asymmetry through the spectral energy distribution of projection weights, demonstrating that concentrated spectra in Query/Key weights induce feature homogeneity, whereas dispersed spectra in Value weights preserve heterogeneity. Then, we introduce KVSlimmer, an efficient algorithm that captures exact Hessian information through a mathematically exact formulation, and derives a closed-form solution utilizing only forward-pass variables, resulting in a gradient-free approach that is both memory- and time-efficient. Extensive experiments across various models and benchmarks demonstrate that KVSlimmer consistently outperforms SOTA methods. For instance, on Llama3.1-8B-Instruct, it improves the LongBench average score by 0.92 while reducing memory costs and latency by 29% and 28%, respectively.Code is available at https://github.com/lianjunl13-sudo/KVSlimmer.