Positive semi-definite embedding for dimensionality reduction and out-of-sample extensions

TL;DR

This paper proposes a novel positive semi-definite kernel-based nonlinear dimensionality reduction method with out-of-sample extension capabilities, demonstrating robustness to outliers and adaptability to data smoothness.

Contribution

It introduces an adaptive, non-linear embedding approach via an infinite-dimensional semi-definite program with a new out-of-sample extension formula.

Findings

More robust to outliers than spectral embedding methods

Provides an explicit out-of-sample extension formula

Applicable under various smoothness assumptions

Abstract

In machine learning or statistics, it is often desirable to reduce the dimensionality of a sample of data points in a high dimensional space $\mathbb{R}^d$. This paper introduces a dimensionality reduction method where the embedding coordinates are the eigenvectors of a positive semi-definite kernel obtained as the solution of an infinite dimensional analogue of a semi-definite program. This embedding is adaptive and non-linear. We discuss this problem both with weak and strong smoothness assumptions about the learned kernel. A main feature of our approach is the existence of an out-of-sample extension formula of the embedding coordinates in both cases. This extrapolation formula yields an extension of the kernel matrix to a data-dependent Mercer kernel function. Our empirical results indicate that this embedding method is more robust with respect to the influence of outliers, compared with a spectral embedding method.