Convergence of Shallow ReLU Networks on Weakly Interacting Data

TL;DR

This paper proves that shallow ReLU networks with logarithmic width can globally converge on high-dimensional, weakly correlated data using gradient flow, with convergence rates depending on data orthogonality.

Contribution

It introduces a high-dimensional analysis leveraging low data correlation to establish global convergence of shallow ReLU networks with logarithmic width.

Findings

Logarithmic width suffices for high-probability convergence.

Convergence rate is exponential in the number of data points.

Phase transition in convergence speed depending on data orthogonality.

Abstract

We analyse the convergence of one-hidden-layer ReLU networks trained by gradient flow on $n$ data points. Our main contribution leverages the high dimensionality of the ambient space, which implies low correlation of the input samples, to demonstrate that a network with width of order $\log(n)$ neurons suffices for global convergence with high probability. Our analysis uses a Polyak-{\L}ojasiewicz viewpoint along the gradient-flow trajectory, which provides an exponential rate of convergence of $\frac{1}{n}$. When the data are exactly orthogonal, we give further refined characterizations of the convergence speed, proving its asymptotic behavior lies between the orders $\frac{1}{n}$ and $\frac{1}{\sqrt{n}}$, and exhibiting a phase-transition phenomenon in the convergence rate, during which it evolves from the lower bound to the upper, and in a relative time of order $\frac{1}{\log(n)}$.