On Empirical Spectral Distributions for Random Tensor Product Models

TL;DR

This paper extends the understanding of empirical spectral distributions for covariance matrices in random tensor product models, demonstrating almost sure convergence under broader conditions than previously established, relevant for complex dependent data structures.

Contribution

It generalizes convergence results of empirical spectral distributions to a wider range of tensor dimensions and dependence structures, beyond the isotropic case.

Findings

Convergence to Marchenko-Pastur law for larger tensor dimensions

Almost sure convergence under subgaussian and symmetric conditions

Applicable to models with dependent features exhibiting concentration

Abstract

In statistics, assuming samples are independent is reasonable. However, this property can fail to hold for the features, a distinction that has led to several lines of work aiming to remove the latter assumption of independence present in the early literature, while preserving the original conclusions. Empirical spectral distributions of covariance matrices are key for understanding the data, and their almost sure convergence is oftentimes desirable. The random tensor product model, $X=(x_{i_1}x_{i_2}...x_{i_d})_{1 \leq i_1<...<i_d \leq n}$ for $x_1,x_2,\hspace{0.05cm}...\hspace{0.05cm},x_n$ i.i.d., introduced by the machine learning community, has a dependence structure for its features far from trivial and has been studied in recent years. When $x_1 \in \mathbb{R}, \mathbb{E}[x_1^4]<\infty, \frac{d}{n^{1/3}}=o(1),$ the empirical spectral distributions of the covariance matrices were proved to converge almost surely to Marchenko-Pastur laws in the random matrix theory regime. This work extends this result to the range $\frac{d}{n^{1/2}}=~o(1)$ when $x_1$ is symmetric with a subgaussian norm slowly growing in $n$ (the aforesaid range arises naturally, and the result failing when $\frac{d}{n^{1/2}} \to \infty$ appears to be a plausible claim) and shows that similarly to the case with independent features, the almost sure convergence holds under more general conditions on the covariance structure than the isotropic case. The latter result provides a means of deriving convergence for empirical spectral distributions of random matrices, applicable to other models as well so long as their entries exhibit a certain degree of concentration.