EconoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Serving

TL;DR

EconoServe is a system designed to optimize multi-resource utilization in large language model serving, significantly increasing throughput and SLO satisfaction while reducing GPU usage by intelligently managing GPU compute and Key-Value Cache resources.

Contribution

It introduces a novel scheduling and resource management system that maximizes GPU and KVC utilization with SLO guarantees, outperforming existing schedulers in efficiency and resource savings.

Findings

Up to 4× throughput increase at the same latency

Up to 91% reduction in job completion time

Up to 78% GPU reduction while maintaining goodput

Abstract

As Large Language Models (LLMs) continue to grow, reducing costs and alleviating GPU demands has become increasingly critical. However, existing schedulers primarily target either GPU compute or Key-Value Cache (KVC) utilization, failing to fully optimize both GPU compute and KVC usage during each iteration or guarantee timely KVC allocations when needed. To address these challenges, we conducted a trace-based experimental analysis and made insightful observations, leading to the design of a system called EconoServe. EconoServe maximizes multi-resource utilization while ensuring service-level objective (SLO) guarantees in LLM serving. To enable adding prompts to a batch to maximize GPU utilization in each iteration, EconoServe maintains separate waiting queues for prompt processing tasks (PTs) and generation tasks (GTs). It batches GTs with the same predicted response lengths (RL) to save scheduling time and allocates KVC space for the predicted RL to avoid KVC allocation failures. It further has a novel KVC pipelining method, allowing sharing allocated but unused KVC space to enhance KVC utilization. In addition, it prioritizes queued requests that occupy more KVC to release KVC earlier and satisfy request service-level-objective (SLO). Experimental results demonstrate that EconoServe increases throughput by up to 4$\times$ with the same level of latency, generates up to 91\% lower job completion time and up to 91\% higher SLO satisfaction ratio compared to vLLM. It also reduces the number of GPUs used in DistServe by up to 78\% while maintaining the same level of goodput.