AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving

TL;DR

AugServe is a novel adaptive request scheduling framework for augmented LLM inference that significantly improves throughput and reduces latency by dynamically optimizing request order and batching based on runtime conditions.

Contribution

It introduces a two-stage adaptive scheduling strategy and dynamic batching mechanism to enhance inference efficiency for augmented LLM services.

Findings

Achieves 4.7x higher effective throughput than vLLM

Reduces time-to-first-token by up to 96.3%

Outperforms existing systems in latency and throughput

Abstract

As augmented large language models (LLMs) with external tools become increasingly popular in web applications, improving augmented LLM inference serving efficiency and optimizing service-level objectives (SLOs) are critical for enhancing user experience. To achieve this, inference systems must maximize request handling within latency constraints, referred to as increasing effective throughput. However, existing systems face two major challenges: (i) reliance on first-come-first-served (FCFS) scheduling causes severe head-of-line blocking, leading to queuing delays exceeding the SLOs for many requests; and (ii) static batch token limit, which fails to adapt to fluctuating loads and hardware conditions. Both of these factors degrade effective throughput and service quality. This paper presents AugServe, an efficient inference framework designed to reduce queueing latency and enhance effective throughput for augmented LLM inference services. The core idea of AugServe is a two-stage adaptive request scheduling strategy. Specifically, AugServe combines the inference features of augmented LLM requests to optimize the order of scheduling decisions (stage I). These decisions are continuously refined with runtime information (stage II), adapting to both request characteristics and system capabilities. In addition, AugServe dynamically adjusts the token batching mechanism based on hardware status and real-time load, further enhancing throughput performance. Experimental results show that AugServe achieves 4.7x and 3.3x higher effective throughput than vLLM and InferCept, while reducing time-to-first-token (TTFT) by up to 96.3% and 95.0%, respectively.