SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference

TL;DR

This paper introduces a hybrid theoretical and empirical approach to optimize compute resource allocation for Prefill-Decode disaggregated LLM inference, ensuring throughput and SLO compliance.

Contribution

It presents a novel methodology combining queuing theory and benchmarking to determine optimal P/D hardware resources under various constraints.

Findings

Accurately predicts optimal P/D resource allocation in real-world scenarios.

Combines theoretical modeling with empirical benchmarking for resource optimization.

Effectively meets throughput and SLO requirements for LLM inference.

Abstract

Prefill-Decode (P/D) disaggregation has emerged as a widely adopted optimization strategy for Large Language Model (LLM) inference. However, there currently exists no well-established methodology for determining the optimal number of P/D hardware resources, subject to constraints on total throughput, service level objectives (SLOs), and request characteristics - specifically input and output lengths. To address this gap, we propose a hybrid approach that combines theoretical modeling with empirical benchmarking. First, we present a theoretical model for calculating P/D resource counts, which is based on total throughput requirements, request input and output lengths, as well as prefill and decode throughput. Then, to obtain the actual prefill and decode throughput under SLO constraints, we model the prefill process using M/M/1 queuing theory, deriving the achieved prefill throughput from the benchmarked maximum prefill throughput and Time-To-First-Token (TTFT). For the decode phase, we determine the decode batch sizes that meet Time-Per-Output-Token (TPOT) requirements and obtain the corresponding decode throughput through empirical measurements. Our experimental results demonstrate that the proposed method can accurately predict optimal P/D resource allocation in real-world LLM inference scenarios.