Staleness-Centric Optimizations for Parallel Diffusion MoE Inference

TL;DR

This paper introduces DICE, a staleness-centric optimization framework for parallel diffusion MoE inference that reduces communication bottlenecks and improves speed with minimal quality loss.

Contribution

DICE combines interweaved parallelism, selective synchronization, and conditional communication to effectively reduce staleness in expert-parallel diffusion models.

Findings

Achieves 1.26x speedup in diffusion inference.

Reduces staleness-related quality degradation.

Provides scalable optimization for MoE-based diffusion models.

Abstract

Mixture-of-Experts-based (MoE-based) diffusion models demonstrate remarkable scalability in high-fidelity image generation, yet their reliance on expert parallelism introduces critical communication bottlenecks. State-of-the-art methods alleviate such overhead in parallel diffusion inference through computation-communication overlapping, termed displaced parallelism. However, we identify that these techniques induce severe *staleness*-the usage of outdated activations from previous timesteps that significantly degrades quality, especially in expert-parallel scenarios. We tackle this fundamental tension and propose DICE, a staleness-centric optimization framework with a three-fold approach: (1) Interweaved Parallelism introduces staggered pipelines, effectively halving step-level staleness for free; (2) Selective Synchronization operates at layer-level and protects layers vulnerable from staled activations; and (3) Conditional Communication, a token-level, training-free method that dynamically adjusts communication frequency based on token importance. Together, these strategies effectively reduce staleness, achieving 1.26x speedup with minimal quality degradation. Empirical results establish DICE as an effective and scalable solution. Our code is publicly available at https://github.com/Cobalt-27/DICE