FUSCO: High-Performance Distributed Data Shuffling via Transformation-Communication Fusion

TL;DR

FUSCO is a novel communication library that significantly improves distributed data shuffling efficiency in large-scale Mixture-of-Experts models by fusing data transformation with communication, reducing training and inference latency.

Contribution

FUSCO introduces a fused data transformation-communication approach tailored for MoE models, achieving substantial speedups over existing libraries.

Findings

Up to 3.84× speedup over NCCL

Reduces training latency by up to 1.39×

Lowers inference latency in first-token generation

Abstract

Large-scale Mixture-of-Experts (MoE) models rely on \emph{expert parallelism} for efficient training and inference, which splits experts across devices and necessitates distributed data shuffling to route each token to its assigned experts. However, existing communication libraries handle this shuffling poorly; its overhead can account for over half of end-to-end runtime. We present FUSCO, an MoE-friendly communication library that achieves efficient and lightweight data shuffling through fused data transformation and communication, based on the key observation that MoE's expert-major data layout conflicts with the device-major layout expected by communication operations. FUSCO captures the fine-grained data layout, which is then interpreted by a pipelined communication engine that performs the required shuffling efficiently along the communication path. Lightweight planning and load-balancing mechanisms complement the engine by eliminating redundant communication and dispersing traffic. Evaluations on representative benchmarks illustrate that FUSCO achieves up to 3.84$\times$ and 2.01$\times$ speedups over NCCL and DeepEP (the state-of-the-art MoE communication library), respectively. In end-to-end MoE tasks, compared to NCCL and DeepEP, FUSCO reduces the training latency by 1.17-1.39$\times$ and 1.10-1.19$\times$, and lowers the first-token generation latency in inference by 1.09-1.25$\times$ and 1.06-1.16$\times$.