Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts

TL;DR

This paper analyzes the convergence rates of maximum likelihood estimation in Gaussian-gated Mixture of Experts models, revealing different behaviors based on parameter settings and introducing novel loss functions for better understanding.

Contribution

It provides the first convergence analysis for MLE in Gaussian-gated MoE models, addressing the interaction of covariates and expert networks with new Voronoi loss functions.

Findings

MLE exhibits different convergence behaviors depending on location parameters.

The solvability of polynomial systems characterizes these behaviors.

Simulation results support the theoretical analysis.

Abstract

Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural networks for heterogeneous data analysis in several applications of machine learning and statistics. Despite its popularity in practice, a satisfactory level of theoretical understanding of the MoE model is far from complete. To shed new light on this problem, we provide a convergence analysis for maximum likelihood estimation (MLE) in the Gaussian-gated MoE model. The main challenge of that analysis comes from the inclusion of covariates in the Gaussian gating functions and expert networks, which leads to their intrinsic interaction via some partial differential equations with respect to their parameters. We tackle these issues by designing novel Voronoi loss functions among parameters to accurately capture the heterogeneity of parameter estimation rates. Our findings reveal that the MLE has distinct behaviors under two complement settings of location parameters of the Gaussian gating functions, namely when all these parameters are non-zero versus when at least one among them vanishes. Notably, these behaviors can be characterized by the solvability of two different systems of polynomial equations. Finally, we conduct a simulation study to empirically verify our theoretical results.