MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts

TL;DR

This paper introduces MomentumSMoE, a new approach that incorporates momentum into Sparse Mixture of Experts to improve training stability and robustness, demonstrated on vision and language tasks.

Contribution

We establish a theoretical connection between SMoE dynamics and gradient descent, then integrate momentum to enhance stability and robustness, applicable to various SMoE models.

Findings

MomentumSMoE is more stable than traditional SMoE.

MomentumSMoE improves robustness against data contamination.

Applicable to vision and language models with minimal additional cost.

Abstract

Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability in deep learning. SMoE has the potential to exponentially increase parameter count while maintaining the efficiency of the model by only activating a small subset of these parameters for a given sample. However, it has been observed that SMoE suffers from unstable training and has difficulty adapting to new distributions, leading to the model's lack of robustness to data contamination. To overcome these limitations, we first establish a connection between the dynamics of the expert representations in SMoEs and gradient descent on a multi-objective optimization problem. Leveraging our framework, we then integrate momentum into SMoE and propose a new family of SMoEs named MomentumSMoE. We theoretically prove and numerically demonstrate that MomentumSMoE is more stable and robust than SMoE. In particular, we verify the advantages of MomentumSMoE over SMoE on a variety of practical tasks including ImageNet-1K object recognition and WikiText-103 language modeling. We demonstrate the applicability of MomentumSMoE to many types of SMoE models, including those in the Sparse MoE model for vision (V-MoE) and the Generalist Language Model (GLaM). We also show that other advanced momentum-based optimization methods, such as Adam, can be easily incorporated into the MomentumSMoE framework for designing new SMoE models with even better performance, almost negligible additional computation cost, and simple implementations.