LLM Query Scheduling with Prefix Reuse and Latency Constraints

TL;DR

This paper introduces a formal framework and a novel scheduling algorithm for large language model inference that leverages prefix reuse to meet strict latency constraints, improving performance in real-world settings.

Contribution

It reveals limitations of existing scheduling strategies, establishes the NP-hardness of the problem, and proposes the $k$-LPM algorithm with theoretical guarantees and empirical validation.

Findings

$k$-LPM improves TTFT performance under realistic traffic.

Existing FCFS and LPM strategies have limitations with latency constraints.

Empirical results show significant TTFT reductions in practical settings.

Abstract

The efficient deployment of large language models (LLMs) in online settings requires optimizing inference performance under stringent latency constraints, particularly the time-to-first-token (TTFT) and time-per-output-token (TPOT). This paper focuses on the query scheduling problem for LLM inference with prefix reuse, a technique that leverages shared prefixes across queries to reduce computational overhead. Our work reveals previously unknown limitations of the existing first-come-first-serve (FCFS) and longest-prefix-match (LPM) scheduling strategies with respect to satisfying latency constraints. We present a formal theoretical framework for LLM query scheduling under RadixAttention, a prefix reuse mechanism that stores and reuses intermediate representations in a radix tree structure. Our analysis establishes the NP-hardness of the scheduling problem with prefix reuse under TTFT constraints and proposes a novel scheduling algorithm, $k$-LPM, which generalizes existing methods by balancing prefix reuse and fairness in query processing. Theoretical guarantees demonstrate that $k$-LPM achieves improved TTFT performance under realistic traffic patterns captured by a data generative model. Empirical evaluations in a realistic serving setting validates our findings, showing significant reductions in P99 TTFT compared to baseline methods.