Hierarchical Mixtures-of-Experts for Exponential Family Regression Models with Generalized Linear Mean Functions: A Survey of Approximation and Consistency Results

TL;DR

This paper analyzes hierarchical mixtures-of-experts models for exponential family regression, demonstrating their approximation capabilities and consistency in inference, with convergence rates depending on predictor dimension.

Contribution

It provides theoretical results on the approximation rates and consistency of HME models for generalized linear mean functions, extending understanding of their statistical properties.

Findings

HME models can approximate true densities at specific rates depending on predictor dimension.

Likelihood-based inference with HME is consistent as sample size and experts increase.

Convergence rates depend on the Sobolev class of the true mean function.

Abstract

We investigate a class of hierarchical mixtures-of-experts (HME) models where exponential family regression models with generalized linear mean functions of the form psi(ga+fx^Tfgb) are mixed. Here psi(...) is the inverse link function. Suppose the true response y follows an exponential family regression model with mean function belonging to a class of smooth functions of the form psi(h(fx)) where h(...)in W_2^infty (a Sobolev class over [0,1]^{s}). It is shown that the HME probability density functions can approximate the true density, at a rate of O(m^{-2/s}) in L_p norm, and at a rate of O(m^{-4/s}) in Kullback-Leibler divergence. These rates can be achieved within the family of HME structures with no more than s-layers, where s is the dimension of the predictor fx. It is also shown that likelihood-based inference based on HME is consistent in recovering the truth, in the sense that as the sample size n and the number of experts m both increase, the mean square error of the predicted mean response goes to zero. Conditions for such results to hold are stated and discussed.