Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models

TL;DR

This paper proves the theoretical approximation capabilities of mixture of experts models for conditional probability densities in Lebesgue spaces, including convergence results and the richness of gating functions.

Contribution

It establishes denseness and convergence properties of MoE models in Lebesgue spaces, expanding understanding of their approximation power.

Findings

MoE models are dense in Lebesgue spaces for compactly supported variables.

Almost uniform convergence is achieved for univariate inputs.

The richness of soft-max and Gaussian gating functions is demonstrated.

Abstract

Mixture of experts (MoE) models are widely applied for conditional probability density estimation problems. We demonstrate the richness of the class of MoE models by proving denseness results in Lebesgue spaces, when inputs and outputs variables are both compactly supported. We further prove an almost uniform convergence result when the input is univariate. Auxiliary lemmas are proved regarding the richness of the soft-max gating function class, and their relationships to the class of Gaussian gating functions.