Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms

TL;DR

This paper analyzes the theoretical learning rates of vector-valued spectral regularization algorithms, confirming the saturation effect in ridge regression and providing minimax optimal bounds for finite sample risk, including infinite-dimensional outputs.

Contribution

It establishes new lower bounds demonstrating the saturation effect and provides minimax optimal upper bounds for a broad class of vector-valued spectral algorithms.

Findings

Confirmed the saturation effect for vector-valued ridge regression.

Derived a novel lower bound on learning rates for these algorithms.

Provided minimax optimal upper bounds for finite sample risk, applicable to infinite-dimensional outputs.

Abstract

We study theoretical properties of a broad class of regularized algorithms with vector-valued output. These spectral algorithms include kernel ridge regression, kernel principal component regression, various implementations of gradient descent and many more. Our contributions are twofold. First, we rigorously confirm the so-called saturation effect for ridge regression with vector-valued output by deriving a novel lower bound on learning rates; this bound is shown to be suboptimal when the smoothness of the regression function exceeds a certain level. Second, we present the upper bound for the finite sample risk general vector-valued spectral algorithms, applicable to both well-specified and misspecified scenarios (where the true regression function lies outside of the hypothesis space) which is minimax optimal in various regimes. All of our results explicitly allow the case of infinite-dimensional output variables, proving consistency of recent practical applications.