On the convergence of gradient descent for two layer neural networks

TL;DR

This paper proves that gradient descent can exponentially converge when training two-layer neural networks for function approximation, with network width independent of data size, indicating strong approximation capabilities without curse of dimensionality.

Contribution

It demonstrates exponential convergence of gradient descent for two-layer networks using generic chaining, independent of training data size, highlighting their approximation power.

Findings

Gradient descent achieves exponential convergence rate.

Network width needed is independent of data size.

Strong approximation ability without curse of dimensionality.

Abstract

It has been shown that gradient descent can yield the zero training loss in the over-parametrized regime (the width of the neural networks is much larger than the number of data points). In this work, combining the ideas of some existing works, we investigate the gradient descent method for training two-layer neural networks for approximating some target continuous functions. By making use the generic chaining technique from probability theory, we show that gradient descent can yield an exponential convergence rate, while the width of the neural networks needed is independent of the size of the training data. The result also implies some strong approximation ability of the two-layer neural networks without curse of dimensionality.