On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensions

TL;DR

This paper analyzes how overparameterization affects the training of shallow neural networks with quadratic activations in high-dimensional Gaussian settings, providing theoretical convergence results and minimal overparameterization thresholds.

Contribution

It derives convergence properties and minimal overparameterization conditions for effective training of shallow neural networks in high dimensions, extending previous theoretical insights.

Findings

Convergence of gradient flow under certain overparameterization levels

Minimal overparameterization needed for strong signal recovery

Numerical validation for general initializations

Abstract

We study the training dynamics of a shallow neural network with quadratic activation functions and quadratic cost in a teacher-student setup. In line with previous works on the same neural architecture, the optimization is performed following the gradient flow on the population risk, where the average over data points is replaced by the expectation over their distribution, assumed to be Gaussian.We first derive convergence properties for the gradient flow and quantify the overparameterization that is necessary to achieve a strong signal recovery. Then, assuming that the teachers and the students at initialization form independent orthonormal families, we derive a high-dimensional limit for the flow and show that the minimal overparameterization is sufficient for strong recovery. We verify by numerical experiments that these results hold for more general initializations.