Sample complexity and effective dimension for regression on manifolds

TL;DR

This paper develops a theoretical framework for regression on manifolds, showing that the effective complexity depends on the manifold dimension and providing minimax-optimal error bounds for kernel regression methods.

Contribution

It introduces a nonasymptotic Weyl law for manifolds and demonstrates that smooth function spaces on manifolds have finite effective dimension, impacting regression analysis.

Findings

Effective dimension scales with manifold dimension.

Kernel regression achieves minimax-optimal error bounds.

Theoretical insights apply to noisy and noiseless data.

Abstract

We consider the theory of regression on a manifold using reproducing kernel Hilbert space methods. Manifold models arise in a wide variety of modern machine learning problems, and our goal is to help understand the effectiveness of various implicit and explicit dimensionality-reduction methods that exploit manifold structure. Our first key contribution is to establish a novel nonasymptotic version of the Weyl law from differential geometry. From this we are able to show that certain spaces of smooth functions on a manifold are effectively finite-dimensional, with a complexity that scales according to the manifold dimension rather than any ambient data dimension. Finally, we show that given (potentially noisy) function values taken uniformly at random over a manifold, a kernel regression estimator (derived from the spectral decomposition of the manifold) yields minimax-optimal error bounds that are controlled by the effective dimension.