Generalization for Least Squares Regression With Simple Spiked Covariances

TL;DR

This paper analyzes the generalization error of simple linear models with spiked covariance structures, deriving explicit formulas in the asymptotic regime and highlighting the impact of eigenvalues and eigenvectors.

Contribution

It provides the first explicit characterization of generalization error for linear models with spiked covariances, extending understanding of spectral effects.

Findings

Eigenvalues and eigenvectors of the spike significantly affect generalization error.

Derived explicit formulas for generalization error in the asymptotic regime.

Analyzed models with simple spiked covariance structures.

Abstract

Random matrix theory has proven to be a valuable tool in analyzing the generalization of linear models. However, the generalization properties of even two-layer neural networks trained by gradient descent remain poorly understood. To understand the generalization performance of such networks, it is crucial to characterize the spectrum of the feature matrix at the hidden layer. Recent work has made progress in this direction by describing the spectrum after a single gradient step, revealing a spiked covariance structure. Yet, the generalization error for linear models with spiked covariances has not been previously determined. This paper addresses this gap by examining two simple models exhibiting spiked covariances. We derive their generalization error in the asymptotic proportional regime. Our analysis demonstrates that the eigenvector and eigenvalue corresponding to the spike significantly influence the generalization error.