Theoretical insights into the optimization landscape of over-parameterized shallow neural networks

TL;DR

This paper analyzes the optimization landscape of over-parameterized shallow neural networks, demonstrating conditions under which global optima can be efficiently found using local search methods, especially with quadratic and differentiable activations.

Contribution

It provides theoretical guarantees for global optimality and convergence of gradient-based methods in over-parameterized shallow neural networks with specific activation functions.

Findings

Quadratic activations lead to favorable optimization landscape properties.

Gradient descent converges linearly to global optima with proper initialization.

Results hold for arbitrary training data and Gaussian input distribution.

Abstract

In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of parameters in the model. We show that with quadratic activations the optimization landscape of training such shallow neural networks has certain favorable characteristics that allow globally optimal models to be found efficiently using a variety of local search heuristics. This result holds for an arbitrary training data of input/output pairs. For differentiable activation functions we also show that gradient descent, when suitably initialized, converges at a linear rate to a globally optimal model. This result focuses on a realizable model where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted weight coefficients.