# Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms

## TL;DR

This paper introduces a novel, globally consistent algorithm for learning Mixture-of-Experts models that overcomes local optima issues, providing theoretical guarantees and superior empirical performance.

## Contribution

The paper presents the first algorithm that directly estimates expert parameters in MoE models with provable global consistency, breaking the deadlock of traditional methods.

## Key findings

- Algorithm successfully recovers true parameters in synthetic data.
- Empirical results show superior performance over standard baselines.
- The method is effective on real-world datasets.

## Abstract

Mixture-of-Experts (MoE) is a widely popular model for ensemble learning and is a basic building block of highly successful modern neural networks as well as a component in Gated Recurrent Units (GRU) and Attention networks. However, present algorithms for learning MoE including the EM algorithm, and gradient descent are known to get stuck in local optima. From a theoretical viewpoint, finding an efficient and provably consistent algorithm to learn the parameters remains a long standing open problem for more than two decades. In this paper, we introduce the first algorithm that learns the true parameters of a MoE model for a wide class of non-linearities with global consistency guarantees. While existing algorithms jointly or iteratively estimate the expert parameters and the gating paramters in the MoE, we propose a novel algorithm that breaks the deadlock and can directly estimate the expert parameters by sensing its echo in a carefully designed cross-moment tensor between the inputs and the output. Once the experts are known, the recovery of gating parameters still requires an EM algorithm; however, we show that the EM algorithm for this simplified problem, unlike the joint EM algorithm, converges to the true parameters. We empirically validate our algorithm on both the synthetic and real data sets in a variety of settings, and show superior performance to standard baselines.