Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA

TL;DR

This paper establishes a theoretical link between Locally Linear Embedding, factor analysis, and probabilistic PCA by analyzing LLE's linear reconstruction step through a stochastic lens and connecting it to probabilistic models.

Contribution

It introduces a stochastic perspective on LLE's linear reconstruction, revealing its theoretical connection to factor analysis and probabilistic PCA, bridging spectral and probabilistic dimensionality reduction methods.

Findings

LLE's linear reconstruction can be viewed as a probabilistic model.

A formal connection between LLE, factor analysis, and probabilistic PCA is established.

LLE is explained as a nonlinear extension of linear probabilistic models.

Abstract

Locally Linear Embedding (LLE) is a nonlinear spectral dimensionality reduction and manifold learning method. It has two main steps which are linear reconstruction and linear embedding of points in the input space and embedding space, respectively. In this work, we look at the linear reconstruction step from a stochastic perspective where it is assumed that every data point is conditioned on its linear reconstruction weights as latent factors. The stochastic linear reconstruction of LLE is solved using expectation maximization. We show that there is a theoretical connection between three fundamental dimensionality reduction methods, i.e., LLE, factor analysis, and probabilistic Principal Component Analysis (PCA). The stochastic linear reconstruction of LLE is formulated similar to the factor analysis and probabilistic PCA. It is also explained why factor analysis and probabilistic PCA are linear and LLE is a nonlinear method. This work combines and makes a bridge between two broad approaches of dimensionality reduction, i.e., the spectral and probabilistic algorithms.