Past-Future Scheduler for LLM Serving under SLA Guarantees

TL;DR

This paper introduces the Past-Future scheduler for LLM serving, which accurately estimates memory needs considering request length distributions, leading to significantly improved throughput under SLA constraints.

Contribution

The paper presents a novel Past-Future scheduler that improves memory estimation accuracy for LLM batching, enabling better throughput and SLA adherence.

Findings

Achieves 2-3x higher goodput than existing schedulers.

Effectively balances request queuing and evictions across diverse scenarios.

Demonstrates superior performance in a new high-performance LLM serving framework.

Abstract

The exploration and application of Large Language Models (LLMs) is thriving. To reduce deployment costs, continuous batching has become an essential feature in current service frameworks. The effectiveness of continuous batching relies on an accurate estimate of the memory requirements of requests. However, due to the diversity in request output lengths, existing frameworks tend to adopt aggressive or conservative schedulers, which often result in significant overestimation or underestimation of memory consumption. Consequently, they suffer from harmful request evictions or prolonged queuing times, failing to achieve satisfactory throughput under strict Service Level Agreement (SLA) guarantees (a.k.a. goodput), across various LLM application scenarios with differing input-output length distributions. To address this issue, we propose a novel Past-Future scheduler that precisely estimates the peak memory resources required by the running batch via considering the historical distribution of request output lengths and calculating memory occupancy at each future time point. It adapts to applications with all types of input-output length distributions, balancing the trade-off between request queuing and harmful evictions, thereby consistently achieving better goodput. Furthermore, to validate the effectiveness of the proposed scheduler, we developed a high-performance LLM serving framework, LightLLM, that implements the Past-Future scheduler. Compared to existing aggressive or conservative schedulers, LightLLM demonstrates superior goodput, achieving up to 2-3$\times$ higher goodput than other schedulers under heavy loads. LightLLM is open source to boost the research in such direction (https://github.com/ModelTC/lightllm).