Multi-stage Flow Scheduling for LLM Serving

TL;DR

This paper introduces MFS, a multi-stage flow scheduling mechanism for LLM serving that improves time-to-first-token SLOs by dynamically prioritizing network flows based on laxity, outperforming existing methods.

Contribution

MFS is a novel, stage-aware scheduling approach that approximates LLF without precise slack knowledge, enhancing LLM serving efficiency.

Findings

MFS improves TTFT SLO attainment by 1.2x to 2.4x.

MFS effectively manages complex multi-stage workflows in LLM serving.

Evaluation on real and simulated systems confirms MFS's superior performance.

Abstract

Meeting stringent Time-To-First-Token (TTFT) requirements is crucial for LLM applications. To improve efficiency, modern LLM serving systems adopt disaggregated architectures with diverse parallelisms, introducing complex multi-stage workflows involving reusable KV-block retrieval, collective communication, and P2D transfer. Flows from dependent stages overlap within and across requests on shared bottleneck links, making TTFT highly susceptible to network contention and necessitating stage-aware scheduling. Unfortunately, most existing works schedule flows in a stage-agnostic manner, leading to uncoordinated contention that constitutes a primary cause of SLO violations. In this paper, we present MFS, a holistic multi-stage flow scheduling mechanism designed to maximize TTFT SLO attainment. At its core, MFS approximates the Least-Laxity-First (LLF) scheduling policy without requiring precise knowledge of a request's remaining slack. It achieves this through a Defer-and-Promote principle implemented through a Reverse Multi-Level Queue (RMLQ) structure. By dynamically promoting task precedence as effective laxity diminishes, MFS prioritizes flows with less laxity while preventing requests with loose SLOs from prematurely consuming network bandwidth. We implement MFS as a pluggable module integrated into vLLM, and evaluate it on a 8-server, 32-GPU testbed as well as through large-scale simulations. Our results demonstrate that MFS effectively outperforms state-of-the-art baselines, improving the TTFT SLO attainment by 1.2x--2.4x.