Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications

TL;DR

This paper systematically benchmarks disaggregated LLM serving, analyzing performance and energy implications of different KV cache transfer methods and optimizations, revealing that benefits depend on load and transfer medium.

Contribution

It provides a comprehensive evaluation of disaggregated LLM serving, including new baselines and analysis of energy-performance trade-offs across transfer mediums and optimization strategies.

Findings

Performance benefits depend on request load and transfer medium.

Disaggregation does not guarantee energy savings due to higher energy consumption.

Stage-wise frequency scaling does not lead to energy savings in disaggregated setups.

Abstract

Different from traditional Large Language Model (LLM) serving that colocates the prefill and decode stages on the same GPU, disaggregated serving dedicates distinct GPUs to prefill and decode workload. Once the prefill GPU completes its task, the KV cache must be transferred to the decode GPU. While existing works have proposed various KV cache transfer paths across different memory and storage tiers, there remains a lack of systematic benchmarking that compares their performance and energy efficiency. Meanwhile, although optimization techniques such as KV cache reuse and frequency scaling have been utilized for disaggregated serving, their performance and energy implications have not been rigorously benchmarked. In this paper, we fill this research gap by re-evaluating prefill-decode disaggregation under different KV transfer mediums and optimization strategies. Specifically, we include a new colocated serving baseline and evaluate disaggregated setups under different KV cache transfer paths. Through GPU profiling using dynamic voltage and frequency scaling (DVFS), we identify and compare the performance-energy Pareto frontiers across all setups to evaluate the potential energy savings enabled by disaggregation. Our results show that performance benefits from prefill-decode disaggregation are not guaranteed and depend on the request load and KV transfer mediums. In addition, stage-wise independent frequency scaling enabled by disaggregation does not lead to energy saving due to inherently higher energy consumption of disaggregated serving.