EMO: Frustratingly Easy Progressive Training of Extendable MoE

TL;DR

EMO introduces a progressive training method for MoE models that expands expert capacity over time, improving efficiency without sacrificing performance.

Contribution

It proposes a simple framework that dynamically grows the expert pool during training, addressing efficiency issues in large-scale MoE models.

Findings

EMO matches fixed-expert performance in large-scale experiments.

EMO reduces training time and GPU costs.

It effectively scales MoE training by progressive expert expansion.

Abstract

Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost.