Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

TL;DR

This paper analyzes network topologies for cost-effective large language model serving with mixture-of-experts architectures, revealing switchless topologies as more economical and efficient than traditional scale-up networks.

Contribution

It provides the first systematic cross-layer comparison of network topologies for MoE LLM serving, highlighting the cost-effectiveness of switchless topologies and over-provisioned link bandwidths.

Findings

Switchless topologies outperform scale-up in cost-effectiveness by 20.6-56.2%.

3D full-mesh topology is Pareto-optimal for performance and cost.

Reducing link bandwidth improves throughput per cost by up to 27%.

Abstract

Mixture-of-experts (MoE) architectures have turned LLM serving into a cluster-scale workload in which communication consumes a considerable portion of LLM serving runtime. This has prompted industry to invest heavily in expensive high-bandwidth scale-up networks. We question whether such costly infrastructure is strictly necessary. We present the first systematic cross-layer analysis of network cost-effectiveness for MoE LLM serving, comparing four representative XPU (e.g., GPU/TPU) topologies (scale-up, scale-out, 3D torus, and 3D full-mesh). We find that lower-cost switchless topologies are more cost-effective than the scale-up topology across all serving scenarios explored, improving cost-effectiveness by 20.6-56.2%. In particular, the 3D full-mesh topology is Pareto-optimal in terms of the performance-cost tradeoff. We also find that current scale-up link bandwidths are over-provisioned: reducing the link bandwidth improves throughput per cost by up to 27%. A forward-looking analysis of upcoming GPU generations indicates that the cost-performance advantage of switchless networks will likely persist.