Sparse Infinite Random Feature Latent Variable Modeling

TL;DR

This paper introduces a Bayesian non-parametric latent variable model with an infinite, sparse latent space, automatically determining the number of dimensions and enabling non-linear modeling with improved performance.

Contribution

It presents a novel sparse, infinite-dimensional latent variable model using Indian buffet process priors, with tractable inference via random Fourier approximation and MCMC.

Findings

Superior test set performance on synthetic, biological, and text datasets

Automatic selection of latent dimensions and sparsity in the model

Flexible application beyond Gaussian observation models

Abstract

We propose a non-linear, Bayesian non-parametric latent variable model where the latent space is assumed to be sparse and infinite dimensional a priori using an Indian buffet process prior. A posteriori, the number of instantiated dimensions in the latent space is guaranteed to be finite. The purpose of placing the Indian buffet process on the latent variables is to: 1.) Automatically and probabilistically select the number of latent dimensions. 2.) Impose sparsity in the latent space, where the Indian buffet process will select which elements are exactly zero. Our proposed model allows for sparse, non-linear latent variable modeling where the number of latent dimensions is selected automatically. Inference is made tractable using the random Fourier approximation and we can easily implement posterior inference through Markov chain Monte Carlo sampling. This approach is amenable to many observation models beyond the Gaussian setting. We demonstrate the utility of our method on a variety of synthetic, biological and text datasets and show that we can obtain superior test set performance compared to previous latent variable models.