Regularized least squares learning with heavy-tailed noise is minimax optimal

TL;DR

This paper proves that regularized least squares in reproducing kernel Hilbert spaces achieves optimal convergence rates even with heavy-tailed noise, demonstrating robustness beyond traditional subexponential assumptions.

Contribution

It establishes minimax optimal excess risk bounds for ridge regression under heavy-tailed noise using a novel Fuk-Nagaev inequality approach.

Findings

Achieves convergence rates previously only known under subexponential noise

Demonstrates robustness of regularized least squares to heavy-tailed noise

Provides theoretical guarantees under standard eigenvalue decay conditions

Abstract

This paper examines the performance of ridge regression in reproducing kernel Hilbert spaces in the presence of noise that exhibits a finite number of higher moments. We establish excess risk bounds consisting of subgaussian and polynomial terms based on the well known integral operator framework. The dominant subgaussian component allows to achieve convergence rates that have previously only been derived under subexponential noise - a prevalent assumption in related work from the last two decades. These rates are optimal under standard eigenvalue decay conditions, demonstrating the asymptotic robustness of regularized least squares against heavy-tailed noise. Our derivations are based on a Fuk-Nagaev inequality for Hilbert-space valued random variables.