Inferring Parameters and Structure of Latent Variable Models by Variational Bayes

TL;DR

This paper introduces a Variational Bayes framework for inferring parameters and structures of latent variable models, providing a non-sampling, analytical approach that improves over traditional methods like EM.

Contribution

The paper presents a novel Variational Bayes algorithm that approximates full posterior distributions for model parameters, structures, and latent variables, avoiding sampling and Hessian computations.

Findings

Applicable to various models including clustering and source separation

Guarantees convergence of the inference algorithm

Produces more accurate posterior estimates than traditional methods

Abstract

Current methods for learning graphical models with latent variables and a fixed structure estimate optimal values for the model parameters. Whereas this approach usually produces overfitting and suboptimal generalization performance, carrying out the Bayesian program of computing the full posterior distributions over the parameters remains a difficult problem. Moreover, learning the structure of models with latent variables, for which the Bayesian approach is crucial, is yet a harder problem. In this paper I present the Variational Bayes framework, which provides a solution to these problems. This approach approximates full posterior distributions over model parameters and structures, as well as latent variables, in an analytical manner without resorting to sampling methods. Unlike in the Laplace approximation, these posteriors are generally non-Gaussian and no Hessian needs to be computed. The resulting algorithm generalizes the standard Expectation Maximization algorithm, and its convergence is guaranteed. I demonstrate that this algorithm can be applied to a large class of models in several domains, including unsupervised clustering and blind source separation.