KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

TL;DR

KVDrive is a multi-tier cache management system that optimizes long-context LLM inference by orchestrating cache placement, pipeline scheduling, and cross-tier coordination, significantly improving throughput.

Contribution

It introduces a systems-level approach to manage cache and data movement across GPU, host DRAM, and SSD for scalable long-context inference.

Findings

Achieves up to 1.74x higher throughput than state-of-the-art.

Effectively manages cache to maximize reuse and reduce data movement.

Eliminates stalls by restructuring the decoding pipeline.

Abstract

Supporting long-context LLMs is challenging due to the substantial memory demands of the key-value (KV) cache. Existing offloading systems store the full cache in host memory and selectively fetch critical entries during decoding, but this strategy quickly hits a ceiling: sparsity cannot be pushed further without degrading accuracy. As a result, when context length and batch size grow, the volume of KV transfers rises sharply and becomes the dominant source of decoding latency. We present KVDrive, a holistic multi-tier KV cache management system spanning GPU memory, host DRAM, and SSD. Unlike prior work that pursues greater sparsity through algorithmic refinements, KVDrive tackles the problem from a systems perspective - jointly orchestrating cache placement, pipeline scheduling, and cross-tier coordination to sustain high-throughput inference under tight GPU budgets. KVDrive advances three fundamental capabilities: it adapts cache management to attention behavior to maximize reuse and minimize redundant data movement; it restructures the decoding pipeline to overlap I/O- and CPU/GPU compute-bound stages, eliminating stalls across heterogeneous resources; and it harmonizes data movement across memory tiers to unlock scalable long-context inference far beyond GPU and DRAM limits. We have implemented a fully functional prototype of KVDrive and evaluated it on long-context benchmarks with popular LLMs. The system achieves up to 1.74x higher throughput compared to state-of-the-art works while preserving accuracy.