Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization

TL;DR

This paper extends Gaussian universality analysis to high-dimensional empirical risk minimization with non-Gaussian data, providing an asymptotic characterization of estimators and clarifying universality limits.

Contribution

It introduces a heuristic extension of the CGMT to non-Gaussian data, deriving an asymptotic min-max characterization of ERM estimators and their Gaussian convolution approximation.

Findings

ERM estimator projections approximate a convolution of non-Gaussian and Gaussian distributions.

Any twice-differentiable regularizer is asymptotically equivalent to a quadratic form.

Numerical simulations validate the theoretical predictions.

Abstract

We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min-Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min-max characterization of key statistics, enabling approximation of the mean $\mu_{\hat{\theta}}$ and covariance $C_{\hat{\theta}}$ of the ERM estimator $\hat{\theta}$. Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate $x$ independent of the training data, the projection $\hat{\theta}^\top x$ approximately follows the convolution of the (generally non-Gaussian) distribution of $\mu_{\hat{\theta}}^\top x$ with an independent centered Gaussian variable of variance $\text{Tr}(C_{\hat{\theta}}\mathbb{E}[xx^\top])$. This result clarifies the scope and limits of Gaussian universality for ERMs. Additionally, we prove that any $\mathcal{C}^2$ regularizer is asymptotically equivalent to a quadratic form determined solely by its Hessian at zero and gradient at $\mu_{\hat{\theta}}$. Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.