# AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving

## TL;DR

AdaptCache introduces a lossy compression system for KV caches in LLM serving, significantly reducing delay and improving quality by optimizing cache storage and retrieval strategies.

## Contribution

It proposes a dynamic, adaptive compression approach for KV caches that balances delay and quality, outperforming static methods in LLM applications.

## Key findings

- Achieves 1.43--2.4x delay savings at same quality
- Attains 6--55% quality improvements at same delay
- Reduces loading delays significantly in large-scale LLM serving

## Abstract

Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing LLM serving systems address such redundant computation by storing the KV caches of processed context and loading the corresponding KV cache when a new request reuses the context. Further, as these LLM applications scale, the total size of KV caches becomes excessively large and requires both DRAM and SSD for full storage. However, prior work that stores KV caches in DRAM and SSD suffers from high loading delays, as most KV cache hits come from SSD, which is slow to load. To increase the KV cache hit rate on DRAM, we identify lossy KV cache compression as a promising approach. We design a lossy compression system that decides the compression algorithm, compression rate and device placement for each KV cache entry to maximise DRAM hits and minimise loading delay without significantly degrading generation quality. Compared to various static compression baselines across three tasks, our system AdaptCache achieves 1.43--2.4 x delay savings at the same quality and 6--55% quality improvements at the same delay.