SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution

TL;DR

This paper presents SMoE, a co-designed algorithm-system approach that reduces memory and latency for Mixture of Experts models on edge devices by expert substitution and efficient scheduling.

Contribution

It introduces expert importance-guided substitution and a reuse-maximizing scheduling policy to improve MoE deployment on resource-constrained edge hardware.

Findings

48% lower decoding latency compared to baseline

Over 60% expert cache hit rate achieved

Maintains nearly lossless accuracy

Abstract

The Mixture of Experts (MoE) architecture has emerged as a key technique for scaling Large Language Models by activating only a subset of experts per query. Deploying MoE on consumer-grade edge hardware, however, is constrained by limited device memory, making dynamic expert offloading essential. Unlike prior work that treats offloading purely as a scheduling problem, we leverage expert importance to guide decisions, substituting low-importance activated experts with functionally similar ones already cached in GPU memory, thereby preserving accuracy. As a result, this design reduces memory usage and data transfer, while largely eliminating PCIe overhead. In addition, we introduce a scheduling policy that maximizes the reuse ratio of GPU-cached experts, further boosting efficiency. Extensive evaluations show that our approach delivers 48% lower decoding latency with over 60% expert cache hit rate, while maintaining nearly lossless accuracy.