Mixture of experts models for multilevel data: modelling framework and approximation theory

TL;DR

This paper introduces a class of mixed mixture of experts (MMoE) models tailored for multilevel data, demonstrating their theoretical capacity to approximate a wide variety of complex data structures and dependence patterns.

Contribution

The paper extends MoE models to multilevel data, proving their density in continuous mixed effects models and their universal approximation capabilities for hierarchical dependence structures.

Findings

MMoE models are dense in the space of continuous mixed effects models.

Nested MMoE models can universally approximate hierarchical dependence structures.

Theoretical results support the flexibility of MMoE in modeling multilevel data.

Abstract

Multilevel data are prevalent in many real-world applications. However, it remains an open research problem to identify and justify a class of models that flexibly capture a wide range of multilevel data. Motivated by the versatility of the mixture of experts (MoE) models in fitting regression data, in this article we extend upon the MoE and study a class of mixed MoE (MMoE) models for multilevel data. Under some regularity conditions, we prove that the MMoE is dense in the space of any continuous mixed effects models in the sense of weak convergence. As a result, the MMoE has a potential to accurately resemble almost all characteristics inherited in multilevel data, including the marginal distributions, dependence structures, regression links, random intercepts and random slopes. In a particular case where the multilevel data is hierarchical, we further show that a nested version of the MMoE universally approximates a broad range of dependence structures of the random effects among different factor levels.