Learning Low-Dimensional Nonlinear Structures from High-Dimensional Noisy Data: An Integral Operator Approach

TL;DR

This paper introduces a kernel-spectral embedding algorithm that effectively learns low-dimensional nonlinear structures from high-dimensional noisy data, with adaptive bandwidth selection and theoretical guarantees on convergence.

Contribution

The method provides a novel, theoretically justified approach for embedding high-dimensional noisy data into low-dimensional manifolds without prior knowledge of the manifold.

Findings

Convergence of embeddings to noiseless counterparts in large samples

Characterization of the effect of noise on convergence rates

Superior empirical performance on real datasets

Abstract

We propose a kernel-spectral embedding algorithm for learning low-dimensional nonlinear structures from high-dimensional and noisy observations, where the datasets are assumed to be sampled from an intrinsically low-dimensional manifold and corrupted by high-dimensional noise. The algorithm employs an adaptive bandwidth selection procedure which does not rely on prior knowledge of the underlying manifold. The obtained low-dimensional embeddings can be further utilized for downstream purposes such as data visualization, clustering and prediction. Our method is theoretically justified and practically interpretable. Specifically, we establish the convergence of the final embeddings to their noiseless counterparts when the dimension and size of the samples are comparably large, and characterize the effect of the signal-to-noise ratio on the rate of convergence and phase transition. We also prove convergence of the embeddings to the eigenfunctions of an integral operator defined by the kernel map of some reproducing kernel Hilbert space capturing the underlying nonlinear structures. Numerical simulations and analysis of three real datasets show the superior empirical performance of the proposed method, compared to many existing methods, on learning various manifolds in diverse applications.