A Precise Performance Analysis of Learning with Random Features

TL;DR

This paper provides an exact asymptotic analysis of learning with random features, characterizing training and generalization errors in various regimes, and highlighting the impact of regularization and activation functions on double descent.

Contribution

It offers the first precise asymptotic characterization of learning performance with random features for general data, activation, and loss functions.

Findings

Asymptotic errors match empirical results even in moderate dimensions.

Regularization and activation functions influence the double descent phenomenon.

The analysis applies to both under- and over-parameterized regimes.

Abstract

We study the problem of learning an unknown function using random feature models. Our main contribution is an exact asymptotic analysis of such learning problems with Gaussian data. Under mild regularity conditions for the feature matrix, we provide an exact characterization of the asymptotic training and generalization errors, valid in both the under-parameterized and over-parameterized regimes. The analysis presented in this paper holds for general families of feature matrices, activation functions, and convex loss functions. Numerical results validate our theoretical predictions, showing that our asymptotic findings are in excellent agreement with the actual performance of the considered learning problem, even in moderate dimensions. Moreover, they reveal an important role played by the regularization, the loss function and the activation function in the mitigation of the "double descent phenomenon" in learning.