PROSERVE: Unified Multi-Priority Request Scheduling for LLM Serving

TL;DR

PROSERVE introduces a unified scheduling framework for LLM serving that optimizes request prioritization and SLO adherence, significantly enhancing system gain and performance in diverse, real-world scenarios.

Contribution

The paper formalizes multi-priority request scheduling as a service gain maximization problem and proposes PROSERVE, a two-tier framework with dynamic batching and gain-aware dispatching.

Findings

Outperforms baselines with up to 35% higher system gain.

Increases SLO attainment by up to 52%.

Effective in diverse datasets and real-world industrial trace.

Abstract

The widespread deployment of large language models (LLMs) for interactive applications necessitates serving systems that can handle thousands of concurrent requests with diverse Service Level Objective (SLO) requirements. A critical yet often overlooked dimension in this context is the inherent priority difference among clients; for instance, business-critical functions demand higher performance guarantees, as fulfilling such requests yields significantly greater business value. However, existing LLM serving schedulers fail to jointly optimize for both SLO attainment and client-level priorities. To bridge this gap, we first \textit{formalize multi-priority request scheduling as a service gain maximization problem}, where satisfying latency requirements for requests of different priorities contributes varying levels of gain. We then propose PROSERVE, a unified two-tier scheduling framework designed to maximize overall service gain. At the engine level, SlideBatching dynamically adapts batch formation and request ordering under varying load conditions, employing a sliding boundary mechanism to balance deadline-first and density-first strategies. At the service level, GoRouting performs gain-oriented and capability-aware dispatching across distributed instances, proactively reserving capacity for future high-priority or long requests. Extensive evaluation across four open-source datasets and a real-world industrial trace demonstrates that \systemname{} consistently outperforms state-of-the-art baselines, improving system gain by up to 35% and boosting SLO attainment by up to 52%.