AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

TL;DR

AdaServe is a novel LLM serving system that uses SLO-customized speculative decoding to efficiently meet diverse latency requirements, significantly reducing violations and increasing throughput.

Contribution

It introduces a hardware-aware, SLO-specific speculative decoding framework that formulates multi-SLO serving as a constrained optimization problem.

Findings

Reduces SLO violations by up to 4.3×

Improves goodput by up to 1.9×

Adapts dynamically to workload variations

Abstract

Modern large language model (LLM) applications exhibit diverse service-level objectives (SLOs), from low-latency requirements in interactive coding assistants to more relaxed constraints in data wrangling tasks. Existing LLM serving systems, which rely on uniform batching and scheduling strategies, often fail to meet these heterogeneous SLOs concurrently. We present AdaServe, the first LLM serving system designed to support efficient multi-SLO serving through SLO-customized speculative decoding. AdaServe formulates multi-SLO serving as a constrained optimization problem and introduces a hardware-aware algorithm that constructs a speculation tree tailored to each request's latency target. It features a speculate-select-verify pipeline that enables fine-grained control over decoding speed while maximizing system throughput. AdaServe further adapts to workload variation by dynamically adjusting speculation parameters. Evaluations across diverse workloads show that AdaServe reduces SLO violations by up to 4.3$\times$ and improves goodput by up to 1.9$\times$ compared to the best performing baselines, highlighting its effectiveness in multi-SLO serving.