MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE

TL;DR

MoNE introduces a novel expert pruning approach that replaces redundant experts with lightweight novices, significantly reducing memory costs while maintaining high model performance across various tasks.

Contribution

This paper presents MoNE, a new expert pruning method that evaluates redundancy and replaces experts with lightweight novices, improving robustness and efficiency in structured pruning of MoE models.

Findings

MoNE outperforms baseline methods with up to 2.72% accuracy gain.

Minimal performance drop of 0.14% on Qwen2-57B-A14B under 25% pruning.

Effective across multiple downstream tasks and model architectures.

Abstract

Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token. However, deploying MoE-based models incurs significant memory overhead due to the need to retain all experts in memory. While structured pruning is promising to reduce memory costs, existing methods often show suboptimal performance and unstable degradation in three dimensions: model architectures, calibration data sources, and calibration sample sizes. This paper proposes Mixture-of-Novices-and-Experts (MoNE), a novel expert pruning method that replaces redundant experts with lightweight novices to achieve effective and robust model compression. MoNE evaluates expert redundancy based on two metrics: access frequency and output variance. Experts exhibiting low usage and stable outputs are pruned and replaced with lightweight novices-unbiased estimations of their original outputs-minimizing performance degradation. Extensive experiments demonstrate that MoNE consistently outperforms baseline methods with minimal accuracy degradation across the three dimensions, confirming its effectiveness and robustness. Notably, it outperforms baselines by up to 2.72 for the average zero shot accuracy across nine downstream tasks under 25% pruning ratio, with only 0.14 performance drop for Qwen2-57B-A14B. The code is available at https://github.com/zxgx/mode-pd.