Swarm: Co-Activation Aware KVCache Offloading Across Multiple SSDs

TL;DR

Swarm is a system that exploits the stable co-activation patterns of KVCache entries in large language models to efficiently offload cache data across multiple SSDs, significantly improving I/O performance.

Contribution

It introduces the concept of KVCache Co-Activation and develops Swarm, a multi-SSD offloading system that enhances parallel I/O and bandwidth utilization for LLM inference workloads.

Findings

Reduces I/O time by 2.41x

Improves bandwidth utilization by 2.72x

Effectively adapts to evolving access patterns

Abstract

The key-value (KV) cache has become the dominant contributor to memory consumption in large language model (LLM) inference. Although offloading KVCache from GPU high-bandwidth memory (HBM) to CPU DRAM alleviates device memory pressure, DRAM remains capacity-limited and costly for large, persistent workloads. Solid-state drives (SSDs) provide a cost-effective alternative, but naive SSD-based paging is fundamentally bandwidth-bound due to limited PCIe throughput and per-device bandwidth constraints. In this paper, we observe that KVCache activations in real-world workloads exhibit strong and stable correlations. We term this phenomenon KVCache Co-Activation, where accessing a KV entry is often accompanied by a stable and recurring set of other KV entries. Leveraging this property, we present Swarm, an SSD-based KVCache offloading system that converts bandwidth-bound single-device access into parallel I/O across multiple SSDs. Specifically, Swarm clusters co-activated KV entries offline and distributes the resulting clusters across SSDs using graph-based placement with selective replication to maximize parallel I/O bandwidth. At runtime, Swarm performs load-balanced cluster retrieval and dynamically adapts clustering and caching decisions to sustain high bandwidth utilization under evolving access patterns. Evaluations show that Swarm reduces I/O time by 2.41x and improves effective bandwidth utilization by 2.72x.