LAPS: A Length-Aware-Prefill LLM Serving System

TL;DR

LAPS is a length-aware LLM serving system that disaggregates requests based on prompt length and employs adaptive batching, significantly reducing latency and SLO violations in heterogeneous multi-turn workloads.

Contribution

LAPS introduces a novel length-aware disaggregation and batching mechanism, optimizing LLM serving for heterogeneous prompt lengths and workload characteristics.

Findings

Reduces prefill latency by over 30%

Decreases SLO violations by 28% in multi-instance deployments

Improves throughput by 35% under high concurrency

Abstract

LAPS identifies and disaggregates requests with different prompt lengths in LLM serving to reduce TTFT latency. While recent systems have decoupled the prefill and decode stages to improve throughput, they still rely on unified scheduling policies that fail to adapt to heterogeneous workload characteristics. We observe that prompt-length variations lead to distinct performance bottlenecks, motivating an adaptive scheduling strategy. LAPS disaggregates multi-turn long-prefill requests from short-prefill ones and introduces a length-aware smart batching mechanism for short-prefill workloads. It adopts a dual-queue design that supports temporal disaggregation on a single prefill instance or spatial disaggregation across multiple instances. For short-prefill batches, a batch waiting window and CUDA Graph-based clustering mitigate interference from heterogeneous computation, reducing batching delay and lowering average latency. In real multi-turn workloads, LAPS reduces prefill latency by over 30\% compared to vanilla SGLang under prefill-decode disaggregation, and further decreases SLO violations by 28\% in multi-instance deployments with vanilla data-parallel configuration. Compared to the SGLang router with load balancing, it further lowers SLO violations by 12\% in multi-GPU settings. Under high concurrency and mixed-request scenarios, LAPS improves request throughput by 35\% serving Qwen2.5-32B model for prefill instance, demonstrating its effectiveness in optimizing heterogeneous LLM serving workloads.