Avoiding unwanted results in locally linear embedding: A new understanding of regularization

TL;DR

This paper reveals that locally linear embedding (LLE) can produce unwanted results without regularization, introduces the concept of projection patterns, and demonstrates how regularization effectively prevents these issues.

Contribution

The paper provides a mathematical understanding of unwanted results in LLE and proposes regularization as a solution to avoid these issues.

Findings

Unregularized LLE can produce projection patterns and bizarre results.

Regularization effectively prevents unwanted results in LLE.

Numerical examples on Swiss roll data illustrate the phenomena.

Abstract

We demonstrate that locally linear embedding (LLE) inherently admits some unwanted results when no regularization is used, even for cases in which regularization is not supposed to be needed in the original algorithm. The existence of one special type of result, which we call ``projection pattern'', is mathematically proved in the situation that an exact local linear relation is achieved in each neighborhood of the data. These special patterns as well as some other bizarre results that may occur in more general situations are shown by numerical examples on the Swiss roll with a hole embedded in a high dimensional space. It is observed that all these bad results can be effectively prevented by using regularization.