Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics

TL;DR

This paper investigates how key-value cache compression affects large language models by analyzing attention dynamics, revealing structural properties, redundancy, and phase transitions that influence model robustness and scalability.

Contribution

It introduces a physics-inspired framework to understand KV compression as a perturbation of attention routing, uncovering structural insights and resilience profiles across architectures.

Findings

Moderate compression causes minimal accuracy loss but reveals redundancy.

A sharp hallucination safety cliff occurs near 90% compression, linked to phase transition.

Different architectures exhibit distinct routing dynamics and resilience profiles.

Abstract

As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark degradation. We argue these evaluations miss a structural issue: attention is not just storage but routing, and retaining KV pairs does not guarantee semantic accessibility. We propose a physics-inspired view of KV compression as a controlled perturbation of token-level routing, distinguishing retention, accessibility, and utilization. Using synthetic tasks probing multi-entity tracking, disambiguation, coreference, and multi-hop reasoning, we find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy; all models exhibit a sharp hallucination safety cliff near 90% compression, correlated with spikes in Global Eviction Ratio (GER), suggesting a phase transition in semantic reachability; and architectures differ in routing dynamics, with LLaMA showing early consensus and late diversification, and Qwen showing funnel-like late convergence, leading to distinct resilience profiles. Beyond erasure, we identify representational rigidity, where excessive head-level consensus collapses routing flexibility despite token survival. These results suggest sparse token-route structures govern compression tolerance, reframing KV compression as a structural probe of attention geometry and linking long-context scalability to sparsity and the lottery ticket hypothesis in self-attention.