Power-Law Spectrum of the Random Feature Model

TL;DR

This paper investigates how the spectral power-law decay of data covariance matrices is preserved or altered after passing through a random feature layer with nonlinear activation, revealing that the power-law structure is largely maintained with minor logarithmic modifications.

Contribution

It provides a rigorous characterization of the eigenvalue spectrum of the random feature covariance for data with power-law spectra, showing preservation of the spectral decay exponent under random nonlinear projections.

Findings

Eigenvalues follow a power-law decay with the same exponent as the input covariance.

Logarithmic corrections depend on the degree of the monomial activation.

The spectral structure is preserved up to polylogarithmic factors.

Abstract

Scaling laws for neural networks, in which the loss decays as a power-law in the number of parameters, data, and compute, depend fundamentally on the spectral structure of the data covariance, with power-law eigenvalue decay appearing ubiquitously in vision and language tasks. A central question is whether this spectral structure is preserved or destroyed when data passes through the basic building block of a neural network: a random linear projection followed by a nonlinear activation. We study this question for the random feature model: given data $x \sim N(0,H)\in \mathbb{R}^v$ where $H$ has $\alpha$-power-law spectrum ($\lambda_j(H ) \asymp j^{-\alpha}$, $\alpha > 1$), a Gaussian sketch matrix $W \in \mathbb{R}^{v\times d}$, and an entrywise monomial $f(y) = y^{p}$, we characterize the eigenvalues of the population random-feature covariance $\mathbb{E}_{x }[\frac{1}{d}f(W^\top x )^{\otimes 2}]$. We prove matching upper and lower bounds: for all $1 \leq j \leq c_1 d \log^{-(p+1)}(d)$, the $j$-th eigenvalue is of order $\left(\log^{p-1}(j+1)/j\right)^{\alpha}$. For $ c_1 d \log^{-(p+1)}(d)\leq j\leq d$, the $j$-th eigenvalue is of order $j^{-\alpha}$ up to a polylog factor. That is, the power-law exponent $\alpha$ is inherited exactly from the input covariance, modified only by a logarithmic correction that depends on the monomial degree $p$. The proof combines a dyadic head-tail decomposition with Wick chaos expansions for higher-order monomials and random matrix concentration inequalities.