DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

TL;DR

DuetServe is a novel GPU-based LLM serving framework that adaptively partitions GPU resources to optimize throughput and latency, effectively balancing prefill and decode phases without interference.

Contribution

It introduces a dynamic, fine-grained SM multiplexing approach that isolates inference phases only when contention affects latency, improving efficiency over existing methods.

Findings

Up to 1.3x throughput improvement

Maintains low latency SLOs during adaptive multiplexing

Effectively balances resource utilization and inference performance

Abstract

Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades time-between-tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives (SLOs). DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU-GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3x while maintaining low generation latency compared to state-of-the-art frameworks.