Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey

TL;DR

This paper provides a comprehensive tutorial and survey of factor analysis, probabilistic PCA, variational inference, and VAEs, detailing their theoretical foundations, derivations, and training methods for dimensionality reduction and generative modeling.

Contribution

It offers an integrated overview of related models, deriving key equations and explaining training procedures, bridging theory and practice in latent variable models.

Findings

Derivation of ELBO and EM algorithms for variational inference.

Closed-form solutions for probabilistic PCA.

Explanation of VAE training via EM and backpropagation.

Abstract

This is a tutorial and survey paper on factor analysis, probabilistic Principal Component Analysis (PCA), variational inference, and Variational Autoencoder (VAE). These methods, which are tightly related, are dimensionality reduction and generative models. They assume that every data point is generated from or caused by a low-dimensional latent factor. By learning the parameters of distribution of latent space, the corresponding low-dimensional factors are found for the sake of dimensionality reduction. For their stochastic and generative behaviour, these models can also be used for generation of new data points in the data space. In this paper, we first start with variational inference where we derive the Evidence Lower Bound (ELBO) and Expectation Maximization (EM) for learning the parameters. Then, we introduce factor analysis, derive its joint and marginal distributions, and work out its EM steps. Probabilistic PCA is then explained, as a special case of factor analysis, and its closed-form solutions are derived. Finally, VAE is explained where the encoder, decoder and sampling from the latent space are introduced. Training VAE using both EM and backpropagation are explained.