Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference

TL;DR

This paper analyzes real-world key-value cache access patterns in large language model inference workloads, highlighting the need for specialized, efficient distributed caching systems to optimize performance and reduce redundancy.

Contribution

It provides an in-depth analysis of KVC access patterns and evaluates existing storage solutions, emphasizing the necessity for tailored caching systems for LLM inference.

Findings

Existing key-value stores lack optimization for KVC prefilling in LLMs.

High cache reusability in RAG and agent workloads impacts cache management strategies.

Efficient distributed caching with optimized metadata management can improve inference scalability and latency.

Abstract

The increasing adoption of large language models (LLMs) with extended context windows necessitates efficient Key-Value Cache (KVC) management to optimize inference performance. Inference workloads like Retrieval-Augmented Generation (RAG) and agents exhibit high cache reusability, making efficient caching critical to reducing redundancy and improving speed. We analyze real-world KVC access patterns using publicly available traces and evaluate commercial key-value stores like Redis and state-of-the-art RDMA-based systems (CHIME [1] and Sherman [2]) for KVC metadata management. Our work demonstrates the lack of tailored storage solution for KVC prefilling, underscores the need for an efficient distributed caching system with optimized metadata management for LLM workloads, and provides insights into designing improved KVC management systems for scalable, low-latency inference.