Sigma-MoE-Tiny Technical Report

TL;DR

Sigma-MoE-Tiny introduces an extremely sparse MoE language model with 96 experts per layer, achieving high efficiency and top-tier performance with only 0.5B activated parameters, and proposes a novel load balancing approach for such sparsity.

Contribution

The paper presents Sigma-MoE-Tiny, a highly sparse MoE model with a new load balancing method, demonstrating effective training and competitive performance at a small parameter footprint.

Findings

Achieves top-tier performance with only 0.5B active parameters.

Introduces a progressive sparsification schedule for load balancing.

Maintains training stability despite extreme sparsity.

Abstract

Mixture-of-Experts (MoE) has emerged as a promising paradigm for foundation models due to its efficient and powerful scalability. In this work, we present Sigma-MoE-Tiny, an MoE language model that achieves the highest sparsity compared to existing open-source models. Sigma-MoE-Tiny employs fine-grained expert segmentation with up to 96 experts per layer, while activating only one expert for each token, resulting in 20B total parameters with just 0.5B activated. The major challenge introduced by such extreme sparsity lies in expert load balancing. We find that the widely-used load balancing loss tends to become ineffective in the lower layers under this setting. To address this issue, we propose a progressive sparsification schedule aiming to balance expert utilization and training stability. Sigma-MoE-Tiny is pre-trained on a diverse and high-quality corpus, followed by post-training to further unlock its capabilities. The entire training process remains remarkably stable, with no occurrence of irrecoverable loss spikes. Comprehensive evaluations reveal that, despite activating only 0.5B parameters, Sigma-MoE-Tiny achieves top-tier performance among counterparts of comparable or significantly larger scale. In addition, we provide an in-depth discussion of load balancing in highly sparse MoE models, offering insights for advancing sparsity in future MoE architectures. Project page: https://qghuxmu.github.io/Sigma-MoE-Tiny Code: https://github.com/microsoft/ltp-megatron-lm