BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching

TL;DR

BATON introduces a dynamic re-batching method for large language model inference that minimizes idle computations and resource usage, significantly improving query processing efficiency.

Contribution

This paper presents BATON, a novel dynamic re-batching scheme that enhances batch-wise inference efficiency for LLMs without additional resource costs.

Findings

BATON achieves up to 1.75x faster query processing compared to Orca.

It reduces idle computations during batch processing.

The method maintains inference correctness while optimizing resource utilization.

Abstract

The advanced capabilities of Large Language Models (LLMs) have inspired the development of various interactive web services or applications, such as ChatGPT, which offer query inference services for users. Unlike traditional DNN model, the inference of LLM entails different iterations of forward computation for different queries, which result in efficiency challenges for existing run-to-completion batch-wise inference. Hence, some methods refine batch-wise inference to iteration-level by duplicating all nonlinear layers of LLM. However, this approach not only increases resource usage but also introduces idle computations to the batch due to the prefilling of newly added queries. Therefore, we propose BATON, an efficient batch-wise LLM inference scheme by dynamically adjusting processing batch, which can achieve near-zero idle computations without incurring additional resource consumption. To do so, BATON 1) shapes the vectors involved in the inference of the newly inserted query and processing batch to align dimensions and generates a new attention mask based on vector shaping to ensure inference correctness, which enables query inserting without consuming additional resource; 2) embeds prefilled Keys and Values of the new query into the KV_Cache of the processing batch by leveraging the prefilling and decoding separation mechanism, eliminating idle computations to the batch introduced by the prefilling process of the new query. Experimental results show that compared to the state-of-the-art solution Orca, BATON improves query processing by up to 1.75 times.