Concentration of Random Feature Matrices in High-Dimensions

TL;DR

This paper analyzes the spectral properties of random feature matrices in high dimensions, showing that their singular values concentrate near their expectation and one, which has implications for the conditioning and generalization of random feature models.

Contribution

The paper provides new concentration results for the singular values of asymmetric random feature matrices under various data and weight configurations.

Findings

Singular values concentrate near their expectation with high probability.

Concentration results hold even in moderate dimensions due to logarithmic dependence.

Numerical experiments verify the theoretical spectral concentration bounds.

Abstract

The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models. Random feature matrices are asymmetric rectangular nonlinear matrices depending on two input variables, the data and the weights, which can make their characterization challenging. We consider two settings for the two input variables, either both are random variables or one is a random variable and the other is well-separated, i.e. there is a minimum distance between points. With conditions on the dimension, the complexity ratio, and the sampling variance, we show that the singular values of these matrices concentrate near their full expectation and near one with high-probability. In particular, since the dimension depends only on the logarithm of the number of random weights or the number of data points, our complexity bounds can be achieved even in moderate dimensions for many practical setting. The theoretical results are verified with numerical experiments.