Gradient Span Algorithms Make Predictable Progress in High Dimension

TL;DR

This paper demonstrates that gradient span algorithms exhibit predictable, deterministic behavior in high-dimensional Gaussian models, explaining why different training runs of large models often have similar cost curves despite randomness.

Contribution

It proves asymptotic determinism of gradient span algorithms on high-dimensional Gaussian functions, providing a theoretical explanation for observed training consistency in large models.

Findings

Gradient span algorithms become deterministic in high dimensions.

Training runs of large models show similar cost curves despite randomness.

The results apply to models like spin glasses and random quadratic functions.

Abstract

We prove that all 'gradient span algorithms' have asymptotically deterministic behavior on scaled Gaussian random functions as the dimension tends to infinity. In particular, this result explains the counterintuitive phenomenon that different training runs of many large machine learning models result in approximately equal cost curves despite random initialization on a complicated non-convex landscape. The distributional assumption of (non-stationary) isotropic Gaussian random functions we use is sufficiently general to serve as realistic model for machine learning training but also encompass spin glasses and random quadratic functions.