On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks

TL;DR

This paper provides a theoretical analysis of mixture-of-experts networks, showing their ability to efficiently model complex, structured tasks with low-dimensionality and sparsity priors, and clarifies the roles of architectural components.

Contribution

It offers the first theoretical insights into the expressive power of MoEs for structured tasks, including approximation capabilities and the impact of architecture choices.

Findings

Shallow MoEs can approximate functions on low-dimensional manifolds.

Deep MoEs can represent exponentially many structured functions.

Analysis guides design choices for MoE architectures.

Abstract

Mixture-of-experts networks (MoEs) have demonstrated remarkable efficiency in modern deep learning. Despite their empirical success, the theoretical foundations underlying their ability to model complex tasks remain poorly understood. In this work, we conduct a systematic study of the expressive power of MoEs in modeling complex tasks with two common structural priors: low-dimensionality and sparsity. For shallow MoEs, we prove that they can efficiently approximate functions supported on low-dimensional manifolds, overcoming the curse of dimensionality. For deep MoEs, we show that $\mathcal{O}(L)$-layer MoEs with $E$ experts per layer can approximate piecewise functions comprising $E^L$ pieces with compositional sparsity, i.e., they can exhibit an exponential number of structured tasks. Our analysis reveals the roles of critical architectural components and hyperparameters in MoEs, including the gating mechanism, expert networks, the number of experts, and the number of layers, and offers natural suggestions for MoE variants.