Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration

TL;DR

This paper introduces a new randomized kernel regression algorithm that achieves optimal error bounds and accelerates convergence in low-noise settings, addressing longstanding theoretical gaps.

Contribution

The paper presents a novel randomized algorithm for kernel ridge regression with optimal generalization bounds and improved rates in low-noise scenarios.

Findings

Achieves optimal generalization error bounds for kernel regression.

Faster convergence rates in low-noise conditions.

Bridges the gap between existing upper and lower bounds.

Abstract

In this paper, we consider the nonparametric least square regression in a Reproducing Kernel Hilbert Space (RKHS). We propose a new randomized algorithm that has optimal generalization error bounds with respect to the square loss, closing a long-standing gap between upper and lower bounds. Moreover, we show that our algorithm has faster finite-time and asymptotic rates on problems where the Bayes risk with respect to the square loss is small. We state our results using standard tools from the theory of least square regression in RKHSs, namely, the decay of the eigenvalues of the associated integral operator and the complexity of the optimal predictor measured through the integral operator.