KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference

TL;DR

KVSwap is a novel software framework that enables efficient long-context inference on local devices by offloading key-value cache data to disk, overcoming memory limitations while maintaining performance.

Contribution

KVSwap introduces a disk-aware KV cache offloading method tailored for embedded and mobile devices, improving memory efficiency and throughput during long-context inference.

Findings

Higher throughput under tight memory budgets

Maintains generation quality compared to existing schemes

Effective utilization of disk storage for KV cache

Abstract

Language models (LMs) underpin emerging mobile and embedded AI applications like meeting and video summarization and document analysis, which often require processing multiple long-context inputs. Running an LM locally on-device improves privacy, enables offline use, and reduces cost, but long-context inference quickly hits a \emph{memory capacity wall} as the key-value (KV) cache grows linearly with context length and batch size. Existing KV-cache offloading schemes are designed to transfer cache data from GPU memory to CPU memory; however, they are not suitable for embedded and mobile systems, where the CPU and GPU (or NPU) typically share a unified memory and the non-volatile secondary storage (disk) offers limited I/O bandwidth. We present KVSwap, a software framework tailored for local devices that achieves high memory efficiency while effectively leveraging disk storage. KVSwap stores the full cache on disk, uses highly compact in-memory metadata to predict which entries to preload, overlaps computation with hardware-aware disk access, and orchestrates read patterns to match storage device characteristics. Our evaluation shows that across representative LMs and storage types, KVSwap delivers higher throughput under tight memory budgets while maintaining generation quality over existing KV cache offloading schemes.