Gaussian Process-Gated Hierarchical Mixtures of Experts

TL;DR

This paper introduces Gaussian process-gated hierarchical mixtures of experts (GPHMEs), a novel model that uses Gaussian processes for gating and experts, offering improved performance, interpretability, and efficiency for large-scale data analysis.

Contribution

The paper presents a new hierarchical mixture of experts model using Gaussian processes for gating and experts, enhancing performance and interpretability over existing methods.

Findings

Outperforms tree-based HME benchmarks in accuracy

Achieves good performance with reduced complexity

Demonstrates effectiveness on large-scale datasets

Abstract

In this paper, we propose novel Gaussian process-gated hierarchical mixtures of experts (GPHMEs). Unlike other mixtures of experts with gating models linear in the input, our model employs gating functions built with Gaussian processes (GPs). These processes are based on random features that are non-linear functions of the inputs. Furthermore, the experts in our model are also constructed with GPs. The optimization of the GPHMEs is performed by variational inference. The proposed GPHMEs have several advantages. They outperform tree-based HME benchmarks that partition the data in the input space, and they achieve good performance with reduced complexity. Another advantage is the interpretability they provide for deep GPs, and more generally, for deep Bayesian neural networks. Our GPHMEs demonstrate excellent performance for large-scale data sets, even with quite modest sizes.