ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor

TL;DR

ELIS introduces an efficient LLM serving system with a novel response length predictor and an optimized scheduling strategy, significantly reducing inference wait times in real-world industrial settings.

Contribution

The paper presents ELIS, a new LLM serving system that combines a response length predictor with an ISRTF scheduler to improve inference efficiency and reduce latency.

Findings

ISRTF reduces average job completion time by up to 19.6%.

ELIS effectively manages inference tasks in production environments.

Response length prediction enhances scheduling accuracy.

Abstract

We propose ELIS, a serving system for Large Language Models (LLMs) featuring an Iterative Shortest Remaining Time First (ISRTF) scheduler designed to efficiently manage inference tasks with the shortest remaining tokens. Current LLM serving systems often employ a first-come-first-served scheduling strategy, which can lead to the "head-of-line blocking" problem. To overcome this limitation, it is necessary to predict LLM inference times and apply a shortest job first scheduling strategy. However, due to the auto-regressive nature of LLMs, predicting the inference latency is challenging. ELIS addresses this challenge by training a response length predictor for LLMs using the BGE model, an encoder-based state-of-the-art model. Additionally, we have devised the ISRTF scheduling strategy, an optimization of shortest remaining time first tailored to existing LLM iteration batching. To evaluate our work in an industrial setting, we simulate streams of requests based on our study of real-world user LLM serving trace records. Furthermore, we implemented ELIS as a cloud-native scheduler system on Kubernetes to evaluate its performance in production environments. Our experimental results demonstrate that ISRTF reduces the average job completion time by up to 19.6%.