Multi-Bin Batching for Increasing LLM Inference Throughput

TL;DR

This paper introduces Multi-Bin Batching, a novel method for grouping LLM inference requests by predicted execution times to improve throughput, supported by theoretical analysis and real-world experiments.

Contribution

It proposes a new batching strategy that groups requests into bins based on predicted durations, enhancing inference throughput for large language models.

Findings

Significant throughput improvements over standard batching methods.

Theoretical proof of throughput optimality of the proposed method.

Validated effectiveness through real-world LLM inference scenarios.

Abstract

As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches.