The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets

TL;DR

This paper investigates universal statistical properties of complex datasets using tools from statistical physics and Random Matrix Theory, revealing common scaling laws and structures that can inform neural network analysis.

Contribution

It demonstrates that real-world and synthetic datasets share universal eigenvalue scaling laws and RMT properties, providing new insights into data structure and sample size requirements.

Findings

Eigenvalue power-law scalings differ between real and uncorrelated data.

Generated correlated data can model real-world eigenvalue behavior.

Real-world datasets and synthetic models share the same RMT universality class.

Abstract

We study universal traits which emerge both in real-world complex datasets, as well as in artificially generated ones. Our approach is to analogize data to a physical system and employ tools from statistical physics and Random Matrix Theory (RMT) to reveal their underlying structure. We focus on the feature-feature covariance matrix, analyzing both its local and global eigenvalue statistics. Our main observations are: (i) The power-law scalings that the bulk of its eigenvalues exhibit are vastly different for uncorrelated normally distributed data compared to real-world data, (ii) this scaling behavior can be completely modeled by generating Gaussian data with long range correlations, (iii) both generated and real-world datasets lie in the same universality class from the RMT perspective, as chaotic rather than integrable systems, (iv) the expected RMT statistical behavior already manifests for empirical covariance matrices at dataset sizes significantly smaller than those conventionally used for real-world training, and can be related to the number of samples required to approximate the population power-law scaling behavior, (v) the Shannon entropy is correlated with local RMT structure and eigenvalues scaling, is substantially smaller in strongly correlated datasets compared to uncorrelated ones, and requires fewer samples to reach the distribution entropy. These findings show that with sufficient sample size, the Gram matrix of natural image datasets can be well approximated by a Wishart random matrix with a simple covariance structure, opening the door to rigorous studies of neural network dynamics and generalization which rely on the data Gram matrix.