NanoFlow: Towards Optimal Large Language Model Serving Throughput

TL;DR

NanoFlow is a novel serving framework that enhances large language model throughput by exploiting intra-device parallelism through input splitting and operation duplication, significantly improving efficiency.

Contribution

NanoFlow introduces a method to overlap heterogeneous resource usage within a device, optimizing LLM serving throughput by automatic nano-batch management and resource allocation.

Findings

Achieves 1.91x throughput boost over state-of-the-art systems.

Reaches 50% to 72% of optimal throughput on popular models.

Effective for models like LLaMA-2-70B and Mixtral 8x7B.

Abstract

Large Language Models (LLMs) have resulted in a surging demand for planet-scale serving systems, where tens of thousands of GPUs continuously serve hundreds of millions of users. Consequently, throughput has emerged as a key metric that determines serving systems' performance. Due to large model sizes and memory-intensive self-attention, LLM serving has been commonly assumed to be memory-bound. Through a detailed analysis, we show that despite having memory-intensive components, end-to-end LLM serving is compute bound for most common workloads and LLMs. Alas, most existing serving engines fall short from optimal compute utilization, because the heterogeneous operations that comprise LLM serving--compute, memory, networking--are executed sequentially within a device. We propose NanoFlow, a novel serving framework that exploits intra-device parallelism, which overlaps the usage of heterogeneous resources within a single device. NanoFlow splits inputs into smaller nano-batches and duplicates operations to operate on each portion independently, enabling overlapping. NanoFlow automatically identifies the number, size, ordering, and GPU resource allocation of nano-batches to minimize the execution time, while considering the interference of concurrent operations. We evaluate NanoFlow's end-to-end serving throughput on several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc. With practical workloads, NanoFlow provides 1.91x throughput boost compared to state-of-the-art serving systems achieving 50% to 72% of optimal throughput across popular models.