BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving

TL;DR

BucketServe is a dynamic batching framework for LLM inference that groups requests by size, reducing padding and memory waste, and adapts in real-time to workload fluctuations to improve throughput and SLO compliance.

Contribution

The paper introduces BucketServe, a novel bucket-based dynamic batching method that optimizes GPU memory usage and adapts to workload changes for efficient LLM serving.

Findings

Achieves up to 3.58x throughput improvement over UELLM.

Handles 1.93x more requests under 80% SLO attainment.

Demonstrates 1.975x higher system load capacity compared to UELLM.

Abstract

Large language models (LLMs) have become increasingly popular in various areas, traditional business gradually shifting from rule-based systems to LLM-based solutions. However, the inference of LLMs is resource-intensive or latency-sensitive, posing significant challenges for serving systems. Existing LLM serving systems often use static or continuous batching strategies, which can lead to inefficient GPU memory utilization and increased latency, especially under heterogeneous workloads. These methods may also struggle to adapt to dynamic workload fluctuations, resulting in suboptimal throughput and potential service level objective (SLO) violations. In this paper, we introduce BucketServe, a bucket-based dynamic batching framework designed to optimize LLM inference performance. By grouping requests into size-homogeneous buckets based on sequence length, BucketServe minimizes padding overhead and optimizes GPU memory usage through real-time batch size adjustments preventing out-of-memory (OOM) errors. It introduces adaptive bucket splitting/merging and priority-aware scheduling to mitigate resource fragmentation and ensure SLO compliance. Experiment shows that BucketServe significantly outperforms UELLM in throughput, achieving up to 3.58x improvement. It can also handle 1.93x more request load under the SLO attainment of 80% compared with DistServe and demonstrates 1.975x higher system load capacity compared to the UELLM.