Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving

TL;DR

Aladdin is a cluster-level scheduler for LLM inference that co-optimizes query placement and resource scaling to meet SLOs efficiently, significantly reducing costs.

Contribution

It introduces a novel SLO-aware scheduling approach that jointly manages query placement and resource scaling for LLM serving at the cluster level.

Findings

Reduces serving costs by up to 71% for the same SLO.

Effectively predicts minimal resources needed for SLO compliance.

Improves resource utilization compared to baseline methods.

Abstract

The demand for large language model (LLM) inference is gradually dominating the artificial intelligence workloads. Therefore, there is an urgent need for cost-efficient inference serving. Existing work focuses on single-worker optimization and lacks consideration of cluster-level management for both inference queries and computing resources. However, placing requests and managing resources without considering the query features easily causes SLO violations or resource underutilization. Providers are forced to allocate extra computing resources to guarantee user experience, leading to additional serving costs. In this paper we introduce Aladdin, a scheduler that co-adaptively places queries and scales computing resources with SLO awareness. For a stream of inference queries, Aladdin first predicts minimal computing resources and the corresponding serving workers' configuration required to fulfill the SLOs for all queries. Then, it places the queries to each serving worker according to the prefill and decode latency models of batched LLM inference to maximize each worker's utilization. Results show that Aladdin reduces the serving cost of a single model by up to 71% for the same SLO level compared with the baselines, which can be millions of dollars per year.