EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inference

TL;DR

EC2MoE introduces an adaptive end-cloud pipeline collaboration framework that significantly improves the scalability and efficiency of Mixture-of-Experts inference across heterogeneous environments.

Contribution

The paper presents a hardware-aware expert selection mechanism and a pipeline optimization strategy for scalable MoE inference in end-cloud settings.

Findings

Increases throughput by 2.2x to 5.1x.

Reduces end-to-end latency by 53% to 67%.

Maintains high accuracy and scalability under dynamic conditions.

Abstract

The Mixture-of-Experts (MoE) paradigm has emerged as a promising solution to scale up model capacity while maintaining inference efficiency. However, deploying MoE models across heterogeneous end-cloud environments poses new challenges in expert scheduling, communication overhead, and resource heterogeneity. In this paper, we propose EC2MoE, an adaptive framework for scalable MoE inference via end-cloud pipeline collaboration. First, we design a hardware-aware lightweight group gate network that enhances expert selection and computational efficiency. By incorporating a hardware-aware local expert selection mechanism, the system adaptively filters candidate experts based on real-time device profiles. A lightweight group gate module then integrates local and global gating outputs to achieve high-quality expert routing with minimal overhead. Second, we develop a pipeline optimization mechanism based on endcloud collaboration to accelerate MoE inference. This includes an encoder-decoder structure based on low-rank compression, which reduces transmission and computation costs. And a route-aware heuristic pipeline scheduling algorithm that dynamically allocates inference stages across devices according to workload and network topology. Extensive experiments show that EC2MoE can increase throughput by 2.2x to 5.1x and reduce end-to-end latency by 53% to 67% while maintaining high accuracy compared to state-of-the-art methods. It also maintains good scalability under dynamic load and network environments.