Simultaneous Feature and Expert Selection within Mixture of Experts

TL;DR

This paper proposes a regularized mixture of experts model that simultaneously performs feature and expert selection using L1 regularization to improve classification in high-dimensional spaces.

Contribution

It introduces a novel regularized MOE model with embedded local feature and expert selection, enhancing specialization in high-dimensional data.

Findings

Method improves expert specialization in high-dimensional data

Embedded feature selection reduces model complexity

Expected to enhance classification accuracy

Abstract

A useful strategy to deal with complex classification scenarios is the "divide and conquer" approach. The mixture of experts (MOE) technique makes use of this strategy by joinly training a set of classifiers, or experts, that are specialized in different regions of the input space. A global model, or gate function, complements the experts by learning a function that weights their relevance in different parts of the input space. Local feature selection appears as an attractive alternative to improve the specialization of experts and gate function, particularly, for the case of high dimensional data. Our main intuition is that particular subsets of dimensions, or subspaces, are usually more appropriate to classify instances located in different regions of the input space. Accordingly, this work contributes with a regularized variant of MoE that incorporates an embedded process for local feature selection using $L1$ regularization, with a simultaneous expert selection. The experiments are still pending.