Hierarchical Routing Mixture of Experts

TL;DR

This paper introduces a hierarchical routing mixture of experts model that uses a binary tree structure with classifiers and regressors to better handle complex, multimodal regression data by jointly partitioning the input-output space.

Contribution

The paper proposes a novel binary tree-structured HRME model with a probabilistic framework and EM algorithm for learning, improving regression on complex multimodal data.

Findings

Outperforms other regression models on various tasks.

Effectively partitions complex data with classifiers and simple regressors.

Demonstrates the benefit of hierarchical routing in regression accuracy.

Abstract

In regression tasks the distribution of the data is often too complex to be fitted by a single model. In contrast, partition-based models are developed where data is divided and fitted by local models. These models partition the input space and do not leverage the input-output dependency of multimodal-distributed data, and strong local models are needed to make good predictions. Addressing these problems, we propose a binary tree-structured hierarchical routing mixture of experts (HRME) model that has classifiers as non-leaf node experts and simple regression models as leaf node experts. The classifier nodes jointly soft-partition the input-output space based on the natural separateness of multimodal data. This enables simple leaf experts to be effective for prediction. Further, we develop a probabilistic framework for the HRME model, and propose a recursive Expectation-Maximization (EM) based algorithm to learn both the tree structure and the expert models. Experiments on a collection of regression tasks validate the effectiveness of our method compared to a variety of other regression models.