MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

TL;DR

MoE-Prefill introduces a novel asynchronous expert weight gathering system that significantly improves efficiency and throughput in MoE prefill serving for large language models, reducing redundant computation and synchronization.

Contribution

It proposes MoE-Prefill, a system that decouples expert placement from activation routing, enabling asynchronous expert weight gathering and improved serving performance.

Findings

Achieves 1.35-1.37x throughput over baseline on real workloads.

Up to 1.59x throughput on synthetic long-context workloads.

Sustains 29.8-36.2% per-GPU FLOPs utilization.

Abstract

Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose MoE-Prefill, a prefill-only serving system whose backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than routing them by activation, and whose frontend co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, MoE-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.