Generalizable Spectral Embedding with an Application to UMAP

TL;DR

This paper introduces Sep-SpectralNet, a spectral embedding method that overcomes limitations in generalizability, scalability, and eigenvector separation, thereby broadening the applicability of spectral embedding techniques like UMAP.

Contribution

Sep-SpectralNet extends SpectralNet with a post-processing step for eigenvector separation, addressing all three major limitations of spectral embedding methods.

Findings

Empirically demonstrates generalization and approximation of spectral embedding

Maintains scalability comparable to SpectralNet

Enables generalizable UMAP visualizations

Abstract

Spectral Embedding (SE) is a popular method for dimensionality reduction, applicable across diverse domains. Nevertheless, its current implementations face three prominent drawbacks which curtail its broader applicability: generalizability (i.e., out-of-sample extension), scalability, and eigenvectors separation. Existing SE implementations often address two of these drawbacks; however, they fall short in addressing the remaining one. In this paper, we introduce Sep-SpectralNet (eigenvector-separated SpectralNet), a SE implementation designed to address all three limitations. Sep-SpectralNet extends SpectralNet with an efficient post-processing step to achieve eigenvectors separation, while ensuring both generalizability and scalability. This method expands the applicability of SE to a wider range of tasks and can enhance its performance in existing applications. We empirically demonstrate Sep-SpectralNet's ability to consistently approximate and generalize SE, while maintaining SpectralNet's scalability. Additionally, we show how Sep-SpectralNet can be leveraged to enable generalizable UMAP visualization. Our codes are publicly available.