Switching Efficiency: A Novel Framework for Dissecting AI Data Center Network Efficiency

TL;DR

This paper introduces the Switching Efficiency Framework, a new metric system for analyzing and diagnosing communication bottlenecks in AI data center networks, guiding future design improvements.

Contribution

It proposes a novel, comprehensive metric framework that links network activity to computational progress and decomposes efficiency into analyzable factors.

Findings

Symmetric, distributed switching aligns with sparse LLM traffic.

All-to-All traffic from Mixture-of-Experts models degrades port utilization.

Design choices like resource allocation and in-network computing improve efficiency.

Abstract

Communication is pivotal in LLM training, and a thorough analysis of the communication efficiency of AI data center (AIDC) network is essential for guiding the design of these capital-intensive clusters. However, conventional metrics are inadequate for such analysis, as they do not directly link network activity to computational progress and lack granularity to diagnose the impact of different network design patterns. To address this, we introduce a metric framework, the Switching Efficiency Framework, whose core metric - Switching Efficiency ($\eta$) - quantifies computationally effective data throughput per unit switching capacity. We further decompose $\eta$ into three factors - Data, Routing Efficiency, and Port Utilization to facilitate analysis of distinct communication bottlenecks. Using this metric framework, we demonstrate how the symmetric, distributed switching of 3D-Torus and the centralized, hierarchical switching of Rail-Optimized architecture align with sparse or imbalanced LLM training traffic, and show that All-to-All traffic from Mixture-of-Experts models severely degrades their port utilization and routing efficiency. Our analysis also demonstrates how key design choices - such as adjusting switching resource allocation, expanding server size, adopting in-network computing, and multi-plane design - positively influence distinct facets of communication efficiency. Ultimately, the Switching Efficiency Framework provides an analytical tool for analyzing efficiency bottlenecks, thereby informing the design of future-generation AIDC networks.