A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length

TL;DR

This paper models the queueing delay in large language model inference by analyzing token distribution, batching strategies, and token limits, providing insights for optimizing inference latency in interactive AI applications.

Contribution

It introduces a queueing theoretic framework for analyzing LLM inference delays considering token limits and batching, enabling optimal parameter selection.

Findings

Enforcing small token limits reduces queueing delay.

Different batching strategies impact inference latency.

Models closely match simulation results.

Abstract

Large language models (LLMs) propel the prosperity of interactive AI applications showcased by ChatGPT that demand timely response of inference services. However, LLM inference is computation intensive and memory intensive, and improper parameter configuration at LLM platforms may exacerbate the inference time. In this paper, we analyze the impact of LLM output token distribution on the inference queueing delay, where the max-token clipping and the batched inference are considered. By formulating an M/G/1 model, we observe that enforcing a maximum output token limit on a very small fraction of inference requests can significantly reduce the queueing delay, and our model facilitates the selection of the optimal limit. For the batch inference, we model the service process as a bulk queue in which the batch processing time is affected by the batch size and the maximum token size inside this batch jointly. The queueing delays of the batching of all buffered requests (dynamic batching), the batching of constant number of requests (fixed batching), and the batching without intra-batch waiting (elastic batching) are derived. Experimental results show that our mathematical models coincide with the event-driven simulations well.