Dimensionality Reduction and Wasserstein Stability for Kernel Regression

TL;DR

This paper investigates the effects of dimensionality reduction on kernel regression accuracy, introducing a Wasserstein stability analysis that leads to convergence rates, especially beneficial in semi-supervised learning.

Contribution

It provides a novel Wasserstein stability result for kernel regression and applies it to analyze PCA-based dimensionality reduction in a two-step regression process.

Findings

Derived a stability bound for kernel regression under Wasserstein perturbations.

Established convergence rates for the two-step PCA and kernel regression procedure.

Demonstrated the approach's usefulness in semi-supervised learning contexts.

Abstract

In a high-dimensional regression framework, we study consequences of the naive two-step procedure where first the dimension of the input variables is reduced and second, the reduced input variables are used to predict the output variable with kernel regression. In order to analyze the resulting regression errors, a novel stability result for kernel regression with respect to the Wasserstein distance is derived. This allows us to bound errors that occur when perturbed input data is used to fit the regression function. We apply the general stability result to principal component analysis (PCA). Exploiting known estimates from the literature on both principal component analysis and kernel regression, we deduce convergence rates for the two-step procedure. The latter turns out to be particularly useful in a semi-supervised setting.