Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing

TL;DR

This paper introduces Expert Threshold routing, a dynamic and load-balanced expert routing method for autoregressive language models that improves efficiency and performance without auxiliary losses.

Contribution

The paper proposes a novel Expert Threshold routing mechanism that enables dynamic computation allocation and load balancing in autoregressive models without relying on auxiliary losses.

Findings

ET achieves lower cross-entropy loss than TC-MoE.

ET reduces token usage by 1.6 times for the same performance.

ET is fully causal and suitable for autoregressive language modeling.

Abstract

Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and requiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving average (EMA) threshold estimated from the global token distribution. At both training and inference, each token is independently routed to an expert if its score exceeds the expert's threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates dependence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B parameters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6$\times$ fewer tokens.