Universality Laws for High-Dimensional Learning with Random Features

TL;DR

This paper establishes a universality theorem showing that high-dimensional learning with random features behaves similarly to a Gaussian model, simplifying analysis and confirming a key conjecture in the field.

Contribution

It proves a Gaussian universality theorem for random feature models, confirming the Gaussian equivalence conjecture and advancing theoretical understanding of high-dimensional learning.

Findings

Random feature models are asymptotically equivalent to Gaussian models in training and generalization.

The proof employs a Lindeberg approach, leave-one-out analysis, and Stein's method.

The result simplifies the analysis of complex neural network models in high dimensions.

Abstract

We prove a universality theorem for learning with random features. Our result shows that, in terms of training and generalization errors, a random feature model with a nonlinear activation function is asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix. This settles a so-called Gaussian equivalence conjecture based on which several recent papers develop their results. Our method for proving the universality theorem builds on the classical Lindeberg approach. Major ingredients of the proof include a leave-one-out analysis for the optimization problem associated with the training process and a central limit theorem, obtained via Stein's method, for weakly correlated random variables.