Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation

TL;DR

This paper proposes a novel heterogeneous inference architecture for large language models that disaggregates attention computation onto memory-optimized devices, significantly improving throughput and efficiency.

Contribution

It introduces model-attention disaggregation, enabling efficient splitting of attention operators across heterogeneous devices, and demonstrates its effectiveness with the Lamina system.

Findings

Lamina achieves 16.1% to 90.1% higher throughput than existing solutions.

Disaggregating attention reduces memory bottlenecks and improves resource utilization.

Communication overhead between devices remains manageable with current networking technologies.

Abstract

Transformer-based large language models (LLMs) exhibit impressive performance in generative tasks but also introduce significant challenges in real-world serving due to inefficient use of the expensive, computation-optimized accelerators. Although disaggregated serving architectures have been proposed to split different phases of LLM inference, the efficiency of decoding phase is still low. This is caused by the varying resource demands of different operators in the transformer-based LLMs. Specifically, the attention operator is memory-intensive, exhibiting a memory access pattern that clashes with the strengths of modern accelerators, especially for long context requests. To enhance the efficiency of LLM decoding, we introduce model-attention disaggregation. This approach leverages a collection of cheap, memory-optimized devices for the attention operator while still utilizing high-end accelerators for other parts of the model. This heterogeneous setup ensures that each component is tailored to its specific workload, maximizing overall performance and cost efficiency. Our comprehensive analysis and experiments confirm the viability of splitting the attention computation over multiple devices. Also, the communication bandwidth required between heterogeneous devices proves to be manageable with prevalent networking technologies. To further validate our theory, we develop and deploy Lamina, an LLM inference system that incorporates model-attention disaggregation in a distributed heterogeneous cluster. Experimental results indicate that Lamina can provide 16.1 ~ 90.1% higher estimated throughput than existing solutions with similar costs.