Random feature approximation for general spectral methods

TL;DR

This paper analyzes the generalization properties of spectral regularization methods combined with random features, providing optimal learning rates and extending previous results for kernel algorithms and neural networks.

Contribution

It offers a comprehensive analysis of spectral regularization with random features, achieving optimal rates and covering broader classes than prior work.

Findings

Achieves optimal learning rates over various regularity classes.

Extends analysis to classes beyond the RKHS.

Improves upon previous results for kernel and neural network methods.

Abstract

Random feature approximation is arguably one of the most popular techniques to speed up kernel methods in large scale algorithms and provides a theoretical approach to the analysis of deep neural networks. We analyze generalization properties for a large class of spectral regularization methods combined with random features, containing kernel methods with implicit regularization such as gradient descent or explicit methods like Tikhonov regularization. For our estimators we obtain optimal learning rates over regularity classes (even for classes that are not included in the reproducing kernel Hilbert space), which are defined through appropriate source conditions. This improves or completes previous results obtained in related settings for specific kernel algorithms.