The Power of Random Features and the Limits of Distribution-Free Gradient Descent

TL;DR

This paper establishes a fundamental link between gradient-based training of neural networks and random feature models, revealing inherent limitations of distribution-free learning and introducing a new complexity measure.

Contribution

It proves that models trained with distribution-free gradient descent can be approximated by random features, and introduces the average probabilistic dimension complexity framework.

Findings

Gradient descent models can be approximated by random features.

Distribution-free learning has fundamental limitations.

Introduces the average probabilistic dimension complexity (adc).

Abstract

We study the relationship between gradient-based optimization of parametric models (e.g., neural networks) and optimization of linear combinations of random features. Our main result shows that if a parametric model can be learned using mini-batch stochastic gradient descent (bSGD) without making assumptions about the data distribution, then with high probability, the target function can also be approximated using a polynomial-sized combination of random features. The size of this combination depends on the number of gradient steps and numerical precision used in the bSGD process. This finding reveals fundamental limitations of distribution-free learning in neural networks trained by gradient descent, highlighting why making assumptions about data distributions is often crucial in practice. Along the way, we also introduce a new theoretical framework called average probabilistic dimension complexity (adc), which extends the probabilistic dimension complexity developed by Kamath et al. (2020). We prove that adc has a polynomial relationship with statistical query dimension, and use this relationship to demonstrate an infinite separation between adc and standard dimension complexity.