Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service

TL;DR

This paper presents Kareto, an optimizer for multi-tiered KV cache storage in LLM services that dynamically balances cost, throughput, and latency, improving performance and efficiency over static configurations.

Contribution

It introduces Kareto, a novel multi-objective optimizer that efficiently approximates the Pareto frontier for heterogeneous storage configurations in LLM caching.

Findings

Kareto effectively adapts to workload variations.

It improves throughput by up to 9.3%.

It reduces latency by up to 58.3%.

Abstract

The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity motivates offloading the KV cache to cheaper external storage tiers. While this expands capacity, it introduces the challenge of dynamically managing heterogeneous storage resources to balance cost, throughput, and latency under varying workloads. We formulate this as a multi-objective optimization problem: identifying the Pareto frontier across these metrics within the storage configuration space. Using a high-fidelity end-to-end simulator, we observe that the objective functions are non-analytic and exhibit complex variable coupling, making the Pareto frontier difficult to approximate analytically. To obtain the frontier, we introduce Kareto, a KV-cache Adaptive REsource managemenT Optimizer. Kareto leverages a diminishing-return-guided pruning method to efficiently navigate the large configuration space and approximate the Pareto frontier. Additionally, it incorporates a fine-grained adaptive tuner that uses eviction policies in tier storage and KV block access patterns for group-specific cache management, improving cache efficiency. Experiments on real-world traces show that Kareto adapts to workload and can identify configurations of better cost efficiency, covering static strategies. Compared to the fixed setup with 1024 GB DRAM, Kareto can improve throughput by up to 9.3%, or reduce latency by up to 58.3%, or lower cost by up to 20.2% under respective optimization objectives.