Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving

TL;DR

This paper introduces slice-level scheduling (SCLS) for LLM serving, which improves throughput and load balancing by slicing generation requests, outperforming traditional sequence-level and iteration-level schedulers.

Contribution

The paper proposes a novel slice-level scheduling method that enhances throughput and load balancing in LLM serving by slicing requests and optimizing batching and offloading strategies.

Findings

SCLS improves throughput by up to 315.8% over existing schedulers.

SCLS effectively mitigates load imbalance across multiple LLM instances.

Experimental results validate the efficiency of the proposed approach.

Abstract

Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation length is generally unpredictable, it is difficult to estimate the time and memory required to process requests, thus posing a challenge for effective request scheduling. Conventional sequence-level scheduling (SLS) serves requests in a first-come first-served (FCFS) manner with static batching where requests with short generation lengths are delayed until those with long ones have finished generation. Besides, to avoid out-of-memory (OOM) errors, SLS batches requests using a small batch size, which limits throughput. Recently proposed iteration-level scheduling (ILS) improves this with continuous batching, timely completing requests and dynamically adding new ones, but often limits the number of parallel-processing requests to OOM errors, thus compromising throughput. Moreover, both SLS and ILS fail to effectively balance workload across multiple LLM instances. To tackle these challenges, we propose slice-level scheduling (SCLS). By splitting the predefined maximal generation length limit into slices and serving batches slice by slice, it provides a precise range of serving time and memory usage for batched requests, laying the foundation for effective scheduling. Experiments confirm that compared with SLS and ILS schedulers, SCLS can improve throughput by up to 315.8% and greatly mitigate load imbalance with proposed batching and offloading algorithms.