Comparative Characterization of KV Cache Management Strategies for LLM Inference

TL;DR

This paper empirically compares three KV cache management frameworks for LLM inference, analyzing their trade-offs in memory and performance across various conditions.

Contribution

It provides a detailed evaluation of vLLM, InfiniGen, and H2O, revealing their strengths and optimal use cases for KV cache strategies.

Findings

Different frameworks excel under varying request rates and model sizes.

Memory consumption and latency trade-offs depend on cache management techniques.

Guidelines for selecting suitable KV cache strategies based on system constraints.

Abstract

Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value vectors at each layer. These caches are essential to minimize redundant computation during autoregressive token generation, lowering computational complexity from quadratic to linear. However, the growth of KV caches has posed significant system-level challenges, particularly as model sizes increase, context lengths grow, and concurrent requests compete for limited memory resources. Even though several recent frameworks for KV cache management have emerged, their comparative trade-offs in memory consumption and inference performance have not been fully understood, especially under varying request sizes and model configurations. In this work, we conduct an empirical study of three state-of-the-art KV cache management frameworks: vLLM, InfiniGen, and H2O. These frameworks employ techniques such as tensor offloading, token eviction heuristics, and speculative scheduling to balance memory usage and performance. We evaluate their performance in terms of a range of metrics such as latency, throughput, and memory usage across a spectrum of key parameters including request rates, model sizes, and sparsity levels. Our results pinpoint the conditions for each framework to perform the best, revealing the most suitable selection and configuration of KV cache strategies under memory and performance constraints.