PreMoE: Proactive Inference for Efficient Mixture-of-Experts

TL;DR

PreMoE is a training-free framework that efficiently compiles sparse Mixture-of-Experts models for specific deployment scenarios using a novel expert utility metric.

Contribution

It introduces Predicted Expert Utility (PEU) for domain-aware expert ranking without retraining, enabling targeted specialization or generalization.

Findings

PreMoE achieves up to 50% sparsity with minimal performance loss.

PEU provides stable expert importance estimation under high sparsity.

Domain-specific specialists and multi-domain generalists can be efficiently compiled.

Abstract

Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization. We introduce PreMoE, a training-free framework that proactively compiles sparse MoE variants for targeted deployment scenarios. At its core is Predicted Expert Utility (PEU), a robust metric for estimating expert importance from router logits through high-confidence threshold filtering and logit transformation, which together stabilize utility estimation under aggressive sparsity. Using PEU scores computed on a small calibration set, PreMoE produces domain-aware expert rankings that can be used to compile either domain-specific specialists or high-efficiency multi-domain generalists, without any retraining. Across MoE models ranging from 30B to 718B parameters, PreMoE achieves up to 50\% sparsity with nearly no performance loss. It further exposes a practical deployment trade-off: specialists maximize in-domain efficiency, while synthesized generalists retain broader cross-domain capability at the same sparsity budget.