Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

TL;DR

Infinite-LLM introduces a system that disaggregates attention layers and uses pooled GPU memory to efficiently serve large language models with extremely long contexts, significantly improving throughput and resource utilization.

Contribution

The paper presents a novel LLM serving system that handles dynamic context lengths by disaggregating attention layers and pooling GPU memory, enabling scalable and efficient inference.

Findings

Achieves 1.35-3.4x throughput improvement over state-of-the-art methods.

Supports context lengths up to 2000K tokens across 32 GPUs.

Demonstrates effective resource management for dynamic attention behavior.

Abstract

Large Language Models (LLMs) demonstrate substantial potential across a diverse array of domains via request serving. However, as trends continue to push for expanding context sizes, the autoregressive nature of LLMs results in highly dynamic behavior of the attention layers, showcasing significant differences in computational characteristics and memory requirements from the non-attention layers. This presents substantial challenges for resource management and performance optimization in service systems. Existing static model parallelism and resource allocation strategies fall short when dealing with this dynamicity. To address the issue, we propose Infinite-LLM, a novel LLM serving system designed to effectively handle dynamic context lengths. Infinite-LLM disaggregates attention layers from an LLM's inference process, facilitating flexible and independent resource scheduling that optimizes computational performance and enhances memory utilization jointly. By leveraging a pooled GPU memory strategy across a cluster, Infinite-LLM not only significantly boosts system throughput but also supports extensive context lengths. Evaluated on a dataset with context lengths ranging from a few to 2000K tokens across a cluster with 32 A100 GPUs, Infinite-LLM demonstrates throughput improvement of 1.35-3.4x compared to state-of-the-art methods, enabling efficient and elastic LLM deployment.