Block: Balancing Load in LLM Serving with Context, Knowledge and Predictive Scheduling

TL;DR

Block is a distributed, predictive scheduling framework for large language model serving that improves load balancing, reduces latency, and enhances capacity by leveraging request context and deterministic inference characteristics.

Contribution

It introduces a fully distributed, stateless, and predictive scheduler that outperforms heuristic methods in LLM serving environments.

Findings

Boosts serving capacity by up to 16.7%

Reduces P99 tail latency by up to 49.5%

Effective across diverse models and workloads

Abstract

This paper presents Block, a distributed scheduling framework designed to optimize load balancing and auto-provisioning across instances in large language model serving frameworks by leveraging contextual information from incoming requests. Unlike popular model serving systems that rely on monolithic and heuristic task schedulers, Block operates as a fully distributed, stateless, and predictive scheduling system to achieve low overhead, reliability, and scalability. It leverages the deterministic and predictable characteristics of LLM inferences, such as host configurations, response lengths, and hardware performance, to make scheduling decisions based on accurately predicted metrics. Evaluation on a 12 GPUs cluster shows that Block significantly outperforms heuristic schedulers, boosting serving capacity by up to 16.7\% and reducing P99 tail latency by up to 49.5\%. These performance gains remain consistent across diverse models, workloads and configurations. Code and data are open-sourced.