Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction

TL;DR

This paper introduces LU-KV, a novel framework for KV cache eviction that optimizes long-term utility across attention heads, significantly reducing cache size and inference latency in large models.

Contribution

The paper proposes LU-KV, a new head-level budget allocation method for KV cache eviction based on marginal utility, improving efficiency over heuristic approaches.

Findings

Achieves 80% reduction in KV cache size

Reduces inference latency and GPU memory footprint

Maintains model performance with minimal degradation

Abstract

Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference. Current KV cache eviction methods typically rely on instantaneous heuristic metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads. However, this overlooks the heterogeneity in predictive fidelity across attention heads. While certain heads prioritize the instantaneous contribution of tokens, others are dedicated to capturing long-horizon utility. In this paper, we propose that optimal budget allocation should be governed by the marginal utility in preserving long-term semantic information. Based on this insight, we propose LU-KV, a novel framework that optimizes head-level budget allocation through a convex-hull relaxation and a marginal-utility-based greedy solver to achieve near-optimal precision. Furthermore, we implement a data-driven offline profiling protocol to facilitate the practical deployment of LU-KV. Extensive evaluations on LongBench and RULER benchmarks demonstrate that LU-KV achieves an 80% reduction in KV cache size with minimal performance degradation, while simultaneously reducing inference latency and GPU memory footprint.