Sparse Bayesian Unsupervised Learning

TL;DR

This paper introduces a Bayesian method for variable selection and clustering in high-dimensional unsupervised learning, utilizing constrained Gaussian mixture models with a sparsity prior, and demonstrates its theoretical optimality and efficient implementation.

Contribution

It develops a novel Bayesian framework for simultaneous variable selection and clustering, with proven optimality and a fast Metropolis-Hastings algorithm for practical use.

Findings

Proves a sparsity oracle inequality for the method.

Demonstrates fast convergence of the algorithm.

Effectively selects the number of clusters and relevant variables.

Abstract

This paper is about variable selection, clustering and estimation in an unsupervised high-dimensional setting. Our approach is based on fitting constrained Gaussian mixture models, where we learn the number of clusters $K$ and the set of relevant variables $S$ using a generalized Bayesian posterior with a sparsity inducing prior. We prove a sparsity oracle inequality which shows that this procedure selects the optimal parameters $K$ and $S$. This procedure is implemented using a Metropolis-Hastings algorithm, based on a clustering-oriented greedy proposal, which makes the convergence to the posterior very fast.