Route Experts by Sequence, not by Token

TL;DR

SeqTopK is a simple, efficient routing method for Mixture-of-Experts models that allocates experts at the sequence level, improving performance especially in high sparsity regimes.

Contribution

It introduces a minimal, learnable, sequence-level expert allocation method that requires little code change and enhances MoE performance without retraining.

Findings

Consistent improvements over TopK and prior adaptive methods.

Significant gains up to 16.9% in high sparsity regimes.

Requires less than 1% overhead and is compatible with pretrained models.

Abstract

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional modules and hyperparameters, often requiring costly retraining from scratch. We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level. By selecting the top $T \cdot K$ experts across all $T$ tokens, SeqTopK enables end-to-end learned dynamic allocation -- assigning more experts to difficult tokens and fewer to easy ones -- while preserving the same overall budget. SeqTopK requires only a few lines of code, adds less than 1% overhead, and remains fully compatible with pretrained MoE models. Experiments across math, coding, law, and writing show consistent improvements over TopK and prior parameter-free adaptive methods, with gains that become substantially larger under higher sparsity (up to 16.9%). These results highlight SeqTopK as a simple, efficient, and scalable routing strategy, particularly well-suited for the extreme sparsity regimes of next-generation LLMs. Code is available at https://github.com/Y-Research-SBU/SeqTopK.