Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution

TL;DR

This paper investigates intra-expert activation sparsity in MoE models, revealing significant sparsity in pre-trained models and demonstrating substantial speedups in model execution without accuracy loss.

Contribution

It uncovers and leverages intra-expert sparsity in existing MoE models, enabling more efficient execution without modifying model training or parameters.

Findings

Up to 90% intra-expert sparsity without accuracy loss.

Achieved 2.5x speedup in MoE layer execution.

Extended vLLM to utilize intra-expert sparsity for faster inference.

Abstract

Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. However, sparsity through finer expert granularity is becoming increasingly difficult to achieve due to fundamental training challenges such as expert collapse and load imbalance. In this work, we explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models. Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models, without any modification to the activation function or model parameters, providing up to 90% sparsity within each expert without significant accuracy loss. We explore intra-expert activation sparsity across eight off-the-shelf MoE models ranging from 1B to 400B parameters, and extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons, on top of its existing optimizations, achieving up to 2.5 times speedup in MoE layer execution and 1.2 times end-to-end speedup compared to the original dense vLLM baseline.