Convergence Rates for Mixture-of-Experts

TL;DR

This paper analyzes the convergence rates of mixture-of-experts models with polynomial regression experts, providing theoretical insights into optimal choices of the number of experts and their complexity for better model performance.

Contribution

It offers a theoretical study on the convergence rates of ME models, revealing how the number of experts and their polynomial degree affect learning efficiency.

Findings

Convergence rate depends on both number of experts and expert complexity.

Certain combinations of experts and polynomial degree optimize convergence.

Results inform optimal expert selection and model complexity balancing.

Abstract

In mixtures-of-experts (ME) model, where a number of submodels (experts) are combined, there have been two longstanding problems: (i) how many experts should be chosen, given the size of the training data? (ii) given the total number of parameters, is it better to use a few very complex experts, or is it better to combine many simple experts? In this paper, we try to provide some insights to these problems through a theoretic study on a ME structure where $m$ experts are mixed, with each expert being related to a polynomial regression model of order $k$. We study the convergence rate of the maximum likelihood estimator (MLE), in terms of how fast the Kullback-Leibler divergence of the estimated density converges to the true density, when the sample size $n$ increases. The convergence rate is found to be dependent on both $m$ and $k$, and certain choices of $m$ and $k$ are found to produce optimal convergence rates. Therefore, these results shed light on the two aforementioned important problems: on how to choose $m$, and on how $m$ and $k$ should be compromised, for achieving good convergence rates.