Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement

TL;DR

This paper introduces Prism, a framework for efficient, latency-optimized distributed MoE inference on heterogeneous edge servers, reducing latency and communication costs through expert placement and dynamic migration.

Contribution

Prism is the first edge-oriented MoE inference framework that optimizes expert placement considering heterogeneous hardware and workload dynamics.

Findings

Prism reduces inference latency by up to 30.6%.

Prism significantly lowers communication costs compared to baselines.

Prism effectively adapts expert distribution to workload changes.

Abstract

The emergence of Mixture-of-Experts (MoE) has transformed the scaling of large language models by enabling vast model capacity through sparse activation. Yet, converting these performance gains into practical edge deployment remains difficult, as the massive memory footprint and communication demands often overwhelm resource-limited environments. While centralized cloud-based solutions are available, they are frequently plagued by prohibitive infrastructure costs, latency issues, and privacy concerns. Moreover, existing edge-oriented optimizations largely overlook the complexities of heterogeneous hardware, focusing instead on isolated or uniform device setups. In response, this paper proposes Prism, an inference framework engineered for collaborative MoE serving across diverse GPU-equipped edge servers. By leveraging the intrinsic sparsity and input locality of MoE workloads, Prism minimizes inter-server communication and optimizes expert placement within diverse resource constraints. The framework integrates an activation-aware placement strategy that balances local request coverage with memory utilization, supplemented by a runtime migration mechanism to adapt expert distribution to dynamic workload changes. Experiments on contemporary MoE models and datasets demonstrate that Prism reduces inference latency by up to 30.6% and significantly lowers communication costs compared to state-of-the-art baselines, confirming the effectiveness of cooperative edge-based MoE serving.