When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression

TL;DR

This paper introduces a diagnostic method for understanding when value-aware key-value cache eviction improves long-context language model inference, focusing on cache decision effects and evidence preservation.

Contribution

It proposes a fixed-contract diagnostic that isolates cache decision factors, enabling analysis of cache compression impacts on model accuracy and evidence retention.

Findings

Probe is positive on 72.6% of positive-margin cells.

Probe is positive on 32.4% of nonpositive-margin cells.

Order of cache management: recover evidence, rank output, preserve evidence.

Abstract

Long-context LLM inference is bottlenecked by the memory and bandwidth cost of reading large KV caches during decoding. KV compression reduces this cost by keeping only part of the cache, but task accuracy alone does not identify why a selector succeeds or fails. A selector can fail at three steps: it may miss the evidence future decoding needs, give high scores to tokens that do not affect the output, or break related evidence when fitting scores into a small cache. We introduce a fixed-contract diagnostic that holds the selector's setup fixed and changes one decision slot at a time. For value ranking, the probe combines a block's attention mass with the estimated output change from removing it. On LongBench across three models and two budgets, the probe is positive on 72.6% of positive-margin cells and 32.4% of nonpositive-margin cells. NeedleBench M-RT at 32k and a RULER 8k check probe support closure under branched retrieval, and a 264-cell sign evaluation separates support recovery and output-value ranking from leverage effects near the boundary. The resulting order is to recover decode-side evidence, rank its output value, and preserve coupled evidence during projection.