UCCL-EP: Portable Expert-Parallel Communication

TL;DR

UCCL-EP is a portable, high-performance expert-parallel communication system that maintains DeepEP-level efficiency across diverse GPU and NIC hardware by replacing GPU-initiated RDMA with a GPU-CPU control channel and emulating ordering semantics.

Contribution

The paper introduces UCCL-EP, a novel portable EP communication system that achieves high performance across heterogeneous hardware platforms by leveraging a GPU-CPU control channel and RDMA emulation techniques.

Findings

Outperforms existing EP solutions by up to 2.1x on EFA platforms.

Achieves comparable performance to DeepEP on NVIDIA-only platforms.

Improves token throughput by up to 40-45% in various benchmarks.

Abstract

Mixture-of-Experts (MoE) workloads rely on expert parallelism (EP) to achieve high GPU efficiency. State-of-the-art EP communication systems such as DeepEP demonstrate strong performance but exhibit poor portability across heterogeneous GPU and NIC platforms. The poor portability is rooted in architecture: GPU-initiated token-level RDMA communication requires tight vertical integration between GPUs and NICs, e.g., GPU writes to NIC driver/MMIO interfaces. We present UCCL-EP, a portable EP communication system that delivers DeepEP-level performance across heterogeneous GPU and NIC hardware. UCCL-EP replaces GPU-initiated RDMA with a high-throughput GPU-CPU control channel: compact token-routing commands are transferred to multithreaded CPU proxies, which then issue GPUDirect RDMA operations on behalf of GPUs. UCCL-EP further emulates various ordering semantics required by specialized EP communication modes using RDMA immediate data, enabling correctness on NICs that lack such ordering, e.g., AWS EFA. We implement UCCL-EP on NVIDIA and AMD GPUs with EFA and Broadcom NICs. On EFA, it outperforms the best existing EP solution by up to $2.1\times$ for dispatch and combine throughput. On NVIDIA-only platform, UCCL-EP achieves comparable performance to the original DeepEP. UCCL-EP also improves token throughput on SGLang by up to 40% on the NVIDIA+EFA platform, and improves DeepSeek-V3 training throughput over the AMD Primus/Megatron-LM framework by up to 45% on a 16-node AMD+Broadcom platform.