Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod

TL;DR

This paper introduces xDeepServe, a novel production serving system for large-scale MoE LLMs on Huawei's CloudMatrix384 SuperPod, emphasizing disaggregation, low latency, and high throughput.

Contribution

It presents a disaggregated execution architecture and communication layer enabling efficient serving of MoE LLMs at scale on SuperPods.

Findings

Achieves 2400 tokens/sec per chip in peak decoding

Supports diverse models including DeepSeek, Kimi, GLM, Qwen, MiniMax

Reduces time-per-output-token to ~50ms

Abstract

Scaled-out MoE LLMs and scaled-up SuperPods create new systems challenges for production Model-as-a-Service (MaaS), requiring disaggregation, low-latency communication, and decentralized serving. This report presents xDeepServe, the production serving system behind Huawei Cloud's MaaS offering on CloudMatrix384, a 48-server SuperPod with 384 Ascend 910C chips connected by a high-bandwidth UB fabric and global shared memory. It serves models including DeepSeek, Kimi, GLM, Qwen, and MiniMax, among others. xDeepServe is built around Transformerless, a disaggregated execution architecture that decomposes transformer inference into modular units -- attention, feedforward, and MoE -- and supports disaggregated Prefill-Decode and MoE-Attention deployments. To enable disaggregation, we develop XCCL, a memory-semantic communication layer providing microsecond-level point-to-point and scalable all-to-all primitives, and we extend FlowServe with decentralized DP groups and techniques to mitigate stragglers and synchronization variance. In a peak decoding configuration, xDeepServe reaches 2400 tokens/s per Ascend 910C chip at ~50ms time-per-output-token (TPOT).