LAPS: A Length-Aware-Prefill LLM Serving System
Jianshu She, Zonghang Li, Hongchao Du, Shangyu Wu, Wenhao Zheng, Eric Xing, Zhengzhong Liu, Huaxiu Yao, Jason Xue, Qirong Ho

TL;DR
LAPS is a length-aware LLM serving system that disaggregates requests based on prompt length and employs adaptive batching, significantly reducing latency and SLO violations in heterogeneous multi-turn workloads.
Contribution
LAPS introduces a novel length-aware disaggregation and batching mechanism, optimizing LLM serving for heterogeneous prompt lengths and workload characteristics.
Findings
Reduces prefill latency by over 30%
Decreases SLO violations by 28% in multi-instance deployments
Improves throughput by 35% under high concurrency
Abstract
LAPS identifies and disaggregates requests with different prompt lengths in LLM serving to reduce TTFT latency. While recent systems have decoupled the prefill and decode stages to improve throughput, they still rely on unified scheduling policies that fail to adapt to heterogeneous workload characteristics. We observe that prompt-length variations lead to distinct performance bottlenecks, motivating an adaptive scheduling strategy. LAPS disaggregates multi-turn long-prefill requests from short-prefill ones and introduces a length-aware smart batching mechanism for short-prefill workloads. It adopts a dual-queue design that supports temporal disaggregation on a single prefill instance or spatial disaggregation across multiple instances. For short-prefill batches, a batch waiting window and CUDA Graph-based clustering mitigate interference from heterogeneous computation, reducing…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCloud Computing and Resource Management · Software-Defined Networks and 5G · Software System Performance and Reliability
