Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism

TL;DR

This paper introduces CDSP and Tetris, a novel system for optimizing large language model inference by fine-grained, stage-specific sequence parallelism, significantly improving latency and capacity in diverse request scenarios.

Contribution

The paper proposes Chunkwise Dynamic Sequence Parallelism (CDSP) and Tetris, enabling flexible, resource-efficient LLM serving with dynamic parallelism regulation and resource fragmentation exploitation.

Findings

Up to 4.35× lower time-to-first-token (TTFT)

Reduces median time-between-tokens (TBT) by 40.1%

Increases max request capacity by 45%

Abstract

With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing state-of-the-art systems tune the sequence parallelism (SP) allocation. However, current dynamic SP allocation lacks flexibility to (1) support stage-specific parallelism requirements in LLM inference, (2) mitigate the global latency degradation from excessive SP allocation, and (3) exploit resource fragments arising from SP size variation. To tackle this problem, we propose Chunkwise Dynamic Sequence Parallelism (CDSP), a fine-grained parallelism strategy that assigns SP sizes across \textit{intra-request} token segments. Based on CDSP, we build Tetris, an LLM serving system that (1) efficiently integrates CDSP into disaggregated cluster to satisfy parallelism heterogeneity, (2) dynamically regulates SP size expansion based on real-time load conditions, and (3) adaptively explores chunking plans to utilize fragmented resources while meeting per-request demands. Compared with state-of-the-art systems, Tetris achieves up to 4.35$\times$ lower time-to-first-token (TTFT) under max sustainable loads, reduces median time-between-tokens (TBT) by up to 40.1\%, and increases the max request capacity by up to 45\%.