Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective

TL;DR

This paper introduces a unified model for KV cache eviction and query routing in LLM inference, proposing algorithms that improve cache hit rates and load balancing, validated by extensive experiments showing significant performance gains.

Contribution

It provides the first unified mathematical framework for KV cache eviction and query routing, integrating randomized eviction with learning-based routing for better LLM inference efficiency.

Findings

Up to 6.92× increase in cache hit rate

Up to 11.96× reduction in latency

Up to 77.4% increase in throughput

Abstract

KV caching is a fundamental technique for accelerating Large Language Model (LLM) inference by reusing key-value (KV) pairs from previous queries, but its effectiveness under limited memory is highly sensitive to the eviction policy. The default Least Recently Used (LRU) eviction algorithm struggles with dynamic online query arrivals, especially in multi-LLM serving scenarios, where balancing query load across workers and maximizing cache hit rate of each worker are inherently conflicting objectives. We give the first unified mathematical model that captures the core trade-offs between KV cache eviction and query routing. Our analysis reveals the theoretical limitations of existing methods and leads to principled algorithms that integrate provably competitive randomized KV cache eviction with learning-based methods to adaptively route queries with evolving patterns, thus balancing query load and cache hit rate. Our theoretical results are validated by extensive experiments across 4 benchmarks and 3 prefix-sharing settings, demonstrating improvements of up to 6.92$\times$ in cache hit rate, 11.96$\times$ reduction in latency, 14.06$\times$ reduction in time-to-first-token (TTFT), and 77.4% increase in throughput over the state-of-the-art methods. Our code is available at https://github.com/fzwark/KVRouting.