Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

TL;DR

This paper demonstrates that properly optimized Mixture-of-Experts models can outperform dense language models when constrained to equal total parameters, compute, and data, validated by extensive experiments.

Contribution

It introduces a novel framework for designing optimal MoE architectures and shows they can surpass dense models under strict resource constraints.

Findings

MoE models with optimal activation rates outperform dense models under equal resources.

Optimal MoE design remains consistent across different model sizes.

Reusing data can mitigate the trade-off between data amount and performance.

Abstract

Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource constraints -- that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its significant practical value and potential. In this paper, we propose a novel perspective and methodological framework to study this question thoroughly. First, we comprehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data resource. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models at 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All model checkpoints are publicly available.