Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spaces

TL;DR

This paper establishes optimal convergence rates for spectral algorithms like ridge regression and principal component regression in Hilbert space regression, advancing theoretical understanding of their performance.

Contribution

It provides the first high-probability convergence analysis with optimal rates for a broad class of spectral algorithms in non-parametric regression over Hilbert spaces.

Findings

Proves optimal convergence rates under capacity and source conditions.

Extends results to almost sure convergence with optimal rates.

Fills theoretical gaps for non-attainable cases in spectral algorithms.

Abstract

In this paper, we study regression problems over a separable Hilbert space with the square loss, covering non-parametric regression over a reproducing kernel Hilbert space. We investigate a class of spectral/regularized algorithms, including ridge regression, principal component regression, and gradient methods. We prove optimal, high-probability convergence results in terms of variants of norms for the studied algorithms, considering a capacity assumption on the hypothesis space and a general source condition on the target function. Consequently, we obtain almost sure convergence results with optimal rates. Our results improve and generalize previous results, filling a theoretical gap for the non-attainable cases.