Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

TL;DR

This paper explores the geometry of the loss landscape for two-layer neural networks, showing how overparameterization leads to well-separated global minima and favorable local convergence properties.

Contribution

It introduces novel techniques to analyze the geometry of global minima and demonstrates local recoverability of two-layer neural networks in overparameterized regimes.

Findings

Global minima become geometrically separated as sample size increases

Gradient flow converges locally with quantifiable rates

Overparameterization enables local recovery of networks

Abstract

Under mild assumptions, we investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. Utilizing novel techniques, we demonstrate: (i) how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows; and (ii) the local convergence properties and rate of gradient flow dynamics. Our results indicate that two-layer neural networks can be locally recovered in the regime of overparameterization.