Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions

TL;DR

This paper demonstrates that neural network training dynamics with complex Gaussian mixture inputs converge to the same behavior as with simple Gaussian inputs, revealing a universality principle in deep learning.

Contribution

It extends the hidden manifold model to Gaussian mixtures, showing that with proper standardization, the dynamics align with Gaussian cases, enhancing theoretical understanding.

Findings

Dynamics converge to Gaussian behavior with proper standardization

Universality holds across diverse structured input distributions

Supports Gaussian assumptions in deep learning theory

Abstract

Analyzing neural network dynamics via stochastic gradient descent (SGD) is crucial to building theoretical foundations for deep learning. Previous work has analyzed structured inputs within the \textit{hidden manifold model}, often under the simplifying assumption of a Gaussian distribution. We extend this framework by modeling inputs as Gaussian mixtures to better represent complex, real-world data. Through empirical and theoretical investigation, we demonstrate that with proper standardization, the learning dynamics converges to the behavior seen in the simple Gaussian case. This finding exhibits a form of universality, where diverse structured distributions yield results consistent with Gaussian assumptions, thereby strengthening the theoretical understanding of deep learning models.