SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models

TL;DR

SERE is a dynamic expert re-routing method that improves batch decoding efficiency in MoE models by reducing active experts based on similarity, achieving up to 2x speedup with minimal quality loss.

Contribution

SERE introduces a novel similarity-based expert re-routing technique for efficient batch decoding in MoE models, avoiding static pruning and enabling dynamic expert skipping.

Findings

Up to 2.0x speedup in decoding time.

Minimal loss in model quality.

Effective in complex reasoning benchmarks.

Abstract

Mixture-of-Experts (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in production serving, MoE models require batch inference to optimize hardware efficiency, which may cause excessive expert activation and thus slow the memory-bound decoding stage. To address the fundamental tension between batch decoding and expert sparsity, we present SERE, a Similarity-based Expert Re-routing method for Efficient batch decoding in MoE models. SERE dynamically reduces the number of active experts in an input-aware manner by re-routing tokens from secondary experts to their most similar primary counterparts. It also leverages similarity patterns to identify and preserve critical experts, thereby preventing capability loss. Notably, SERE avoids static expert pruning or merging, instead enabling dynamic expert skipping based on batch-level expert redundancy. Additionally, we provide an efficient custom CUDA kernel for SERE, enabling plug-and-play use in vLLM with only a single-line code change. Extensive experiments on various complex reasoning benchmarks demonstrate that SERE achieves up to 2.0x speedup with minimal quality loss, providing a practical solution for cost-efficient and latency-sensitive large-scale MoE deployment. Code implementation of SERE can be found in https://github.com/JL-Cheng/SERE.