From Attention to Disaggregation: Tracing the Evolution of LLM Inference

TL;DR

This paper discusses the shift towards disaggregated inference architectures for large language models, aiming to optimize latency, throughput, and cost by applying distributed systems principles to overcome traditional GPU cluster limitations.

Contribution

It introduces a novel architectural approach that decouples inference phases into scalable components, improving performance and resource utilization for LLM deployment.

Findings

Decoupling inference phases reduces resource contention.

Disaggregated architecture improves latency and throughput.

Independent optimization of inference components enhances efficiency.

Abstract

The evolution of Large Language Models from the Transformer architecture to models with trillions of parameters has shifted the primary bottleneck from model training to real time inference. Deploying these massive models is a complex distributed systems challenge constrained by memory bandwidth, computational throughput, and latency requirements. LLM inference fundamentally requires solving a multi objective optimization problem to minimize latency, maximize throughput, and reduce cost. This paper explores the necessary architectural shift towards disaggregated inference, which applies distributed systems principles such as service decomposition, resource disaggregation, and workload partitioning to overcome the limitations of traditional monolithic GPU clusters. By decoupling the compute intensive prefill phase from the memory intensive decode phase into independently scalable components, this paradigm mitigates resource contention and enables independent optimization of key metrics like Time to First Token and Inter Token Latency.