Relating tSNE and UMAP to Classical Dimensionality Reduction

TL;DR

This paper explores the connections between modern gradient-based dimensionality reduction methods like UMAP and classical techniques such as PCA, MDS, ISOMAP, and LLE, revealing their underlying similarities and interpretability.

Contribution

It demonstrates that classical DR methods can be recovered within the UMAP framework and shows how LLE can produce UMAP-like embeddings with minimal modifications.

Findings

Classical DR methods can be reconstructed using UMAP's attraction and repulsion mechanisms.

Modified LLE can produce embeddings indistinguishable from UMAP outputs.

The UMAP objective is closely related to a modified version of LLE.

Abstract

It has become standard to use gradient-based dimensionality reduction (DR) methods like tSNE and UMAP when explaining what AI models have learned. This makes sense: these methods are fast, robust, and have an uncanny ability to find semantic patterns in high-dimensional data without supervision. Despite this, gradient-based DR methods lack the most important quality that an explainability method should possess: themselves being explainable. That is, given a UMAP output, it is currently unclear what one can say about the corresponding input. We work towards closing this question by relating UMAP to classical DR techniques. Specifically, we show that one can fully recover methods like PCA, MDS, and ISOMAP in the modern DR paradigm: by applying attractions and repulsions onto a randomly initialized dataset. We also show that, with a small change, Locally Linear Embeddings (LLE) can indistinguishably reproduce UMAP outputs. This implies that the UMAP effective objective is minimized by this modified version of LLE (and vice versa). Given this, we discuss what must be true of UMAP emebddings and present avenues for future work.