A Universal Approximation Theorem for Mixture of Experts Models

TL;DR

This paper establishes that mixture of experts models can approximate any continuous function on compact domains, extending their theoretical foundation as universal approximators beyond previous Sobolev space assumptions.

Contribution

The paper proves a universal approximation theorem for MoE models, showing their density in the space of all continuous functions on arbitrary compact domains.

Findings

MoE models are dense in continuous functions on compact domains.

The result extends the universal approximation property to arbitrary compact sets.

Provides theoretical support for the broad applicability of MoE models.

Abstract

The mixture of experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from Sobolev space that is sufficiently differentiable and that the domain of estimation is a compact unit hypercube. We provide an alternative result, which shows that the class of MoE mean functions is dense in the class of all continuous functions over arbitrary compact domains of estimation. Our result can be viewed as a universal approximation theorem for MoE models.