Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models

TL;DR

This paper introduces an $ ext{L}_1$-regularized maximum likelihood estimation method for Gaussian-gated Mixture of Experts models, enhancing feature selection and sparsity in high-dimensional clustering and regression tasks.

Contribution

It proposes an EM-Lasso algorithm for regularized parameter estimation and a BIC-like criterion for model selection in high-dimensional MoE models.

Findings

Regularized MLE outperforms standard MLE in simulations.

The method effectively encourages sparsity in high-dimensional settings.

Model selection via BIC-like criterion is successful.

Abstract

Mixtures-of-Experts models and their maximum likelihood estimation (MLE) via the EM algorithm have been thoroughly studied in the statistics and machine learning literature. They are subject of a growing investigation in the context of modeling with high-dimensional predictors with regularized MLE. We examine MoE with Gaussian gating network, for clustering and regression, and propose an $\ell_1$-regularized MLE to encourage sparse models and deal with the high-dimensional setting. We develop an EM-Lasso algorithm to perform parameter estimation and utilize a BIC-like criterion to select the model parameters, including the sparsity tuning hyperparameters. Experiments conducted on simulated data show the good performance of the proposed regularized MLE compared to the standard MLE with the EM algorithm.