Foundations of Coupled Nonlinear Dimensionality Reduction

TL;DR

This paper introduces the concept of coupled nonlinear dimensionality reduction, providing new generalization bounds and an algorithm, with theoretical guarantees applicable to various kernel-based learning scenarios.

Contribution

It presents the first learning guarantees for coupled dimensionality reduction, analyzing Rademacher complexity and proposing a structural risk minimization algorithm.

Findings

Established upper bounds on Rademacher complexity involving Ky-Fan r-norm

Provided both upper and lower bounds justified by the hypothesis set

Developed an algorithm based on theoretical analysis for coupled manifold and separation function fitting

Abstract

In this paper we introduce and analyze the learning scenario of \emph{coupled nonlinear dimensionality reduction}, which combines two major steps of machine learning pipeline: projection onto a manifold and subsequent supervised learning. First, we present new generalization bounds for this scenario and, second, we introduce an algorithm that follows from these bounds. The generalization error bound is based on a careful analysis of the empirical Rademacher complexity of the relevant hypothesis set. In particular, we show an upper bound on the Rademacher complexity that is in $\widetilde O(\sqrt{\Lambda_{(r)}/m})$, where $m$ is the sample size and $\Lambda_{(r)}$ the upper bound on the Ky-Fan $r$-norm of the associated kernel matrix. We give both upper and lower bound guarantees in terms of that Ky-Fan $r$-norm, which strongly justifies the definition of our hypothesis set. To the best of our knowledge, these are the first learning guarantees for the problem of coupled dimensionality reduction. Our analysis and learning guarantees further apply to several special cases, such as that of using a fixed kernel with supervised dimensionality reduction or that of unsupervised learning of a kernel for dimensionality reduction followed by a supervised learning algorithm. Based on theoretical analysis, we suggest a structural risk minimization algorithm consisting of the coupled fitting of a low dimensional manifold and a separation function on that manifold.