Quadratic number of nodes is sufficient to learn a dataset via gradient descent

TL;DR

This paper proves that a quadratic number of neurons in a two-layer neural network suffices for gradient descent to efficiently find global minima, improving previous bounds under mild activation function conditions.

Contribution

It establishes a new, lower threshold for the number of neurons needed for efficient learning, advancing theoretical understanding of neural network training.

Findings

Gradient descent finds global minima in linear time with quadratic neurons.

The threshold for neuron count is improved over previous results.

The bound is likely optimal given the current method.

Abstract

We prove that if an activation function satisfies some mild conditions and number of neurons in a two-layered fully connected neural network with this activation function is beyond a certain threshold, then gradient descent on quadratic loss function finds the optimal weights of input layer for global minima in linear time. This threshold value is an improvement over previously obtained values. We hypothesise that this bound cannot be improved by the method we are using in this work.