Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference

TL;DR

This paper introduces Kairos, an SLO-aware scheduling system for disaggregated LLM inference that improves service-level adherence and throughput by dynamically prioritizing requests and adaptively batching decode requests.

Contribution

Kairos is a novel scheduling system that combines urgency-based prefill prioritization and slack-guided batching to optimize LLM inference performance.

Findings

Kairos improves TTFT SLO attainment by up to 23.9%.

Kairos increases TPOT SLO attainment by up to 27.1%.

Kairos enhances end-to-end SLO attainment by up to 33.8%.

Abstract

In production environments, large language model (LLM) serving is required to meet stringent service-level objectives (SLOs) amid highly variable request patterns. In practice, request lengths follow a long-tail distribution, which gives rise to head-of-line blocking on the prefill side and underutilization caused by stragglers on the decode side in disaggregated serving architectures. Current systems, which adopt first-come-first-served (FCFS) scheduling for prefill and continuous batching for decode, lack the ability to adapt to this imbalance, resulting in compromised SLO attainment and reduced throughput. To address these challenges, we propose Kairos, an SLO-aware scheduling system equipped with two complementary mechanisms. On the prefill side, Kairos employs urgency-based priority scheduling: it predicts prefill completion times and dynamically selects requests to maximize the attainment of time-to-first-token (TTFT) SLOs. On the decode side, Kairos introduces slack-guided adaptive batching, which leverages the gap between per-step decode time and the time-per-output-token (TPOT) SLO to greedily pack short requests. This approach maximizes throughput while strictly adhering to SLO requirements. We implement Kairos and conduct evaluations using an online serving dataset and a state-of-the-art LLM. Experimental results demonstrate that, compared with state-of-the-art baselines, Kairos improves TTFT SLO attainment by up to 23.9\%, TPOT SLO attainment by up to 27.1\%, end-to-end SLO attainment by up to 33.8\%, and decode throughput by up to 19.3\%.