MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

TL;DR

MoE-Prism introduces a co-design approach that transforms monolithic MoE experts into fine-grained, elastic components, enabling dynamic adaptation to diverse service requirements and significantly improving resource efficiency and performance.

Contribution

It presents a novel two-phase methodology for deconstructing monolithic experts into sub-experts and deploying QoS-aware scheduling for elastic MoE services, without retraining.

Findings

Over 4 times more stable operating points achieved.

Up to 19.9% throughput improvement under latency constraints.

Up to 10.36% latency reduction with resource limits.

Abstract

Mixture-of-Experts (MoE) models, the state-of-the-art in large-scale AI, achieve high quality by sparsely activating parameters. However, their reliance on routing between a few monolithic experts via a top-k mechanism creates a "quality cliff", offering only a few coarse-grained operating points. This inflexibility forces a difficult trade-off between cost and quality, preventing adaptation to diverse Service Level Objectives (SLOs) and leading to significant resource over-provisioning. This paper introduces MoE-Prism, a model-system co-design that transforms rigid MoE models into elastic services. Our methodology is divided into two phases. First, an \emph{Offline Refactoring Engine} systematically deconstructs monolithic experts into fine-grained "sub-experts." This engine employs a partitioning optimization solver that uses a metaheuristic-based approach to group neurons, preserving functional locality without requiring retraining. Second, an \emph{Online Scheduling Engine} leverages this new elasticity through QoS-aware scheduling. It implements specialized policies to solve complex system problems, including maximizing throughput in cloud deployments and managing latency-optimized offloading for memory-constrained devices. Our evaluation across three different MoE models shows that MoE-Prismprovides over 4 times more distinct, stable operating points than the baseline. This allows an AI service to dynamically improve throughput by up to 19.9\% under a strict latency budget or reduce latency by up to 10.36\% under limited resources. MoE-Prism provides the critical "control knob" to bridge the model-system gap, enabling the next generation of adaptive, efficient, and QoS-aware AI services.