Excitation: Momentum For Experts

TL;DR

Excitation is a novel optimizer framework that dynamically modulates updates based on expert utilization in sparse models, improving training stability and performance in Mixture-of-Experts architectures.

Contribution

It introduces a new dynamic update modulation method that enhances training stability and performance in sparse, expert-based models, addressing structural confusion issues.

Findings

Accelerates convergence in language and vision MoE models

Rescues models from structural confusion during training

Consistently improves final performance across tasks

Abstract

We propose Excitation, a novel optimization framework designed to accelerate learning in sparse architectures such as Mixture-of-Experts (MoEs). Unlike traditional optimizers that treat all parameters uniformly, Excitation dynamically modulates updates using batch-level expert utilization. It introduces a competitive update dynamic that amplifies updates to highly-utilized experts and can selectively suppress low-utilization ones, effectively sharpening routing specialization. Notably, we identify a phenomenon of "structural confusion" in deep MoEs, where standard optimizers fail to establish functional signal paths; Excitation acts as a specialization catalyst, "rescuing" these models and enabling stable training where baselines remain trapped. Excitation is optimizer-, domain-, and model-agnostic, requires minimal integration effort, and introduces neither additional per-parameter optimizer state nor learnable parameters, making it highly viable for memory-constrained settings. Across language and vision tasks, Excitation consistently improves convergence speed and final performance in MoE models, indicating that active update modulation is a key mechanism for effective conditional computation.