A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints

TL;DR

This paper develops a queueing-theoretic framework to analyze and predict the stability of large language model inference systems constrained by GPU memory and computation, aiding efficient deployment.

Contribution

It introduces the first explicit model incorporating GPU memory constraints into LLM inference stability analysis, providing practical tools for system deployment and resource planning.

Findings

The framework accurately predicts stability conditions within 10% in real GPU environments.

Derived conditions help determine the necessary cluster size to prevent unbounded queue growth.

Validation shows the model's predictions align closely with actual system behavior.

Abstract

The rapid adoption of large language models (LLMs) has created significant challenges for efficient inference at scale. Unlike traditional workloads, LLM inference is constrained by both computation and the memory overhead of key-value (KV) caching, which accelerates decoding but quickly exhausts GPU memory. In this paper, we introduce the first queueing-theoretic framework that explicitly incorporates both computation and GPU memory constraints into the analysis of LLM inference. Based on this framework, we derive rigorous stability and instability conditions that determine whether an LLM inference service can sustain incoming demand without unbounded queue growth. This result offers a powerful tool for system deployment, potentially addressing the core challenge of GPU provisioning. By combining an estimated request arrival rate with our derived stable service rate, operators can calculate the necessary cluster size to avoid both costly over-purchasing and performance-violating under-provisioning. We further validate our theoretical predictions through extensive experiments in real GPU production environments. Our results show that the predicted stability conditions are highly accurate, with deviations typically within 10%.