When does Gaussian equivalence fail and how to fix it: Non-universal behavior of random features with quadratic scaling

TL;DR

This paper investigates the limitations of Gaussian equivalence theory in high-dimensional random feature models, especially under quadratic scaling, and introduces a Conditional Gaussian Equivalent model to accurately predict behavior when GET fails.

Contribution

It identifies scenarios where GET fails and proposes a new CGE model that captures the true asymptotics in quadratic scaling regimes for RF models.

Findings

CGE model accurately predicts RF behavior when GET fails.

GET yields incorrect predictions for low-dimensional target functions.

Derived sharp asymptotics for training and test errors.

Abstract

A major effort in modern high-dimensional statistics has been devoted to the analysis of linear predictors trained on nonlinear feature embeddings via empirical risk minimization (ERM). Gaussian equivalence theory (GET) has emerged as a powerful universality principle in this context: it states that the behavior of high-dimensional, complex features can be captured by Gaussian surrogates, which are more amenable to analysis. Despite its remarkable successes, numerical experiments show that this equivalence can fail even for simple embeddings -- such as polynomial maps -- under general scaling regimes. We investigate this breakdown in the setting of random feature (RF) models in the quadratic scaling regime, where both the number of features and the sample size grow quadratically with the data dimension. We show that when the target function depends on a low-dimensional projection of the data, such as generalized linear models, GET yields incorrect predictions. To capture the correct asymptotics, we introduce a Conditional Gaussian Equivalent (CGE) model, which can be viewed as appending a low-dimensional non-Gaussian component to an otherwise high-dimensional Gaussian model. This hybrid model retains the tractability of the Gaussian framework and accurately describes RF models in the quadratic scaling regime. We derive sharp asymptotics for the training and test errors in this setting, which continue to agree with numerical simulations even when GET fails. Our analysis combines general results on CLT for Wiener chaos expansions and a careful two-phase Lindeberg swapping argument. Beyond RF models and quadratic scaling, our work hints at a rich landscape of universality phenomena in high-dimensional ERM.