RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation

TL;DR

RAPID-Serve introduces a GPU-based technique that concurrently executes prefill and decode phases of LLM inference, significantly improving throughput and resource utilization under latency constraints.

Contribution

It presents a novel method for simultaneous prefill and decode execution on GPUs, along with adaptive resource management, to enhance LLM inference efficiency.

Findings

Up to 4.1x throughput improvement over existing methods.

Achieves 32x higher throughput under SLO constraints.

Effective resource utilization in resource-constrained environments.

Abstract

Two widely adopted techniques for LLM inference serving systems today are hybrid batching and disaggregated serving. A hybrid batch combines prefill and decode tokens of different requests in the same batch to improve resource utilization and throughput at the cost of increased latency per token. In contrast, disaggregated serving decouples compute-bound prefill and bandwidth-bound decode phases to optimize for service level objectives (SLOs) at the cost of resource under-utilization and KV-cache transfer overheads. To address the limitations of these techniques, we propose RAPID-Serve: a technique to concurrently execute prefill and decode on the same GPU(s) to meet latency SLOs while maintaining high throughput and efficient resource utilization. Furthermore, we propose Adaptive Resource Management for runtime compute resource allocation, optionally leveraging CU masking (a fine-grained Compute Unit partitioning feature on AMD Instinct\textsuperscript{TM} GPUs). RAPID-Serve provides up to 4.1x (average 1.7x) unconstrained throughput improvement and 32x and higher (average 4.9x) throughput improvement under SLO constraints, showing it as an effective strategy compared to the state-of-the-art approaches, particularly in resource-constrained environments.