The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts

TL;DR

This paper demonstrates that increasing the granularity of active experts in Mixture-of-Experts models exponentially enhances their expressivity, supported by theoretical proofs and experimental validation.

Contribution

It provides a theoretical proof of exponential expressivity gains from higher granularity in MoE architectures, complemented by experimental evidence.

Findings

Higher granularity leads to exponentially greater expressivity.

Models with more active experts outperform those with fewer.

Experimental results confirm theoretical predictions.

Abstract

Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total parameter count. This paper investigates the impact of the number of active experts, termed granularity, comparing architectures with many (e.g., 8 per layer in DeepSeek) to those with fewer (e.g., 1 per layer in Llama-4 models). We prove an exponential separation in network expressivity based on this design parameter, suggesting that models benefit from higher granularity. Experimental results corroborate our theoretical findings and illustrate this separation.