LMetric: Simple is Better - Multiplication May Be All You Need for LLM Request Scheduling

TL;DR

This paper introduces a simple multiplication-based scheduling score for large language model requests that effectively balances workload and KV$ acceleration without hyperparameter tuning, outperforming existing methods.

Contribution

The paper proposes a novel, hyperparameter-free multiplication approach for LLM request scheduling that simplifies and improves upon complex existing combinator-based methods.

Findings

Reduces TTFT by up to 92% and 52%.

Reduces TPOT by approximately 20%.

Effective across diverse real-world workloads.

Abstract

High-quality LLM request scheduling requires achieving two key objectives: whether the routed instance has KV$ to accelerate the request execution and whether the workload is balanced across instances. Achieving both objectives is challenging because pursuing one objective may compromise the other. Current approaches adopt various combinators (e.g., linear combinations) to compute a scheduling score combining indicators for the two objectives, which are complex in that they either require significant workload-specific hyperparameter tuning or model-hardware-aware simulator development, and could still lead to suboptimal performance. In this paper, we show that using a simple multiplication of two carefully chosen indicators-one for KV$-aware (new prefill tokens if routed to an instance) and one for load balancing-aware (current batch size of the instance)-as the scheduling score can simultaneously achieve both objectives well without any hyperparameter tuning. The key idea is that the multiplied score considers both objectives in a manner similar to a linear combination, with a nice property that the original hyperparameters are canceled out during comparison so we don't need tuning to find the best parameters. The two indicators are chosen based on our analysis of LLM characteristics, and our extensive experiments show that this simple approach can reduce TTFT by 92% and 52%, and TPOT by 21% and 20%, compared to vLLM-v1 and a production scheduler on real-world workloads covering chatbots, API calls, and coding agents. We also mathematically derive the conditions under which multiplication may fail, and find that such conditions are extremely rare in practice and can be detected (and mitigated) beforehand.