Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism

TL;DR

This paper introduces FinDEP, a fine-grained task scheduling algorithm for disaggregated expert parallelism in MoE models, significantly improving inference throughput by optimizing task overlap and scheduling.

Contribution

FinDEP is the first to support shared experts and variable granularity scheduling in disaggregated MoE inference, enhancing performance and efficiency.

Findings

FinDEP achieves up to 1.61x throughput improvement over prior methods.

FinDEP provides up to 1.24x speedup on a 32-GPU system.

Experiments demonstrate significant performance gains across multiple GPU systems.

Abstract

The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert activation. Recent disaggregated expert parallelism (DEP) distributes attention and experts to dedicated GPU groups but lacks support for shared experts and efficient task scheduling, limiting performance. We propose FinDEP, a fine-grained task scheduling algorithm for DEP that maximizes task overlap to improve MoE inference throughput. FinDEP introduces three innovations: 1) partitioning computation/communication into smaller tasks for fine-grained pipelining, 2) formulating a scheduling optimization supporting variable granularity and ordering, and 3) developing an efficient solver for this large search space. Experiments on four GPU systems with DeepSeek-V2 and Qwen3-MoE show FinDEP improves throughput by up to 1.61x over prior methods, achieving up to 1.24x speedup on a 32-GPU system.