ALISE: Accelerating Large Language Model Serving with Speculative Scheduling

TL;DR

ALISE is a novel LLM inference framework that uses speculative scheduling and adaptive memory management to significantly improve throughput and reduce latency compared to existing solutions.

Contribution

ALISE introduces a speculative scheduler with execution time estimation and a priority-based memory management protocol for efficient LLM serving.

Findings

Up to 1.8x throughput improvement on Alpaca dataset

Up to 2.1x throughput improvement on ShareGPT dataset

Reduces queuing delays and mitigates memory overhead in LLM inference serving

Abstract

Large Language Models (LLMs) represent a revolutionary advancement in the contemporary landscape of artificial general intelligence (AGI). As exemplified by ChatGPT, LLM-based applications necessitate minimal response latency and maximal throughput for inference serving. However, due to the unpredictability of LLM execution, the first-come-first-serve (FCFS) scheduling policy employed by current LLM serving systems suffers from head-of-line (HoL) blocking issues and long job response times. In this paper, we propose a new efficient LLM inference serving framework, named ALISE. The key design paradigm of ALISE is to leverage a novel speculative scheduler by estimating the execution time for each job and exploiting such prior knowledge to assign appropriate job priority orders, thus minimizing potential queuing delays for heterogeneous workloads. Furthermore, to mitigate the memory overhead of the intermediate key-value (KV) cache, we employ a priority-based adaptive memory management protocol and quantization-based compression techniques. Evaluations demonstrate that in comparison to the state-of-the-art solution vLLM, ALISE improves the throughput of inference serving by up to 1.8x and 2.1x under the same latency constraint on the Alpaca and ShareGPT datasets, respectively.