Learning One-hidden-layer ReLU Networks via Gradient Descent

TL;DR

This paper proves that tensor initialization followed by gradient descent can efficiently learn one-hidden-layer ReLU neural networks with multiple neurons, providing the first theoretical guarantees for such practical learning scenarios.

Contribution

It offers the first theoretical analysis of gradient descent convergence for learning multi-neuron ReLU networks with empirical risk minimization.

Findings

Tensor initialization plus gradient descent converges linearly to true parameters.

The method achieves recovery guarantees with statistical error bounds.

Numerical experiments confirm the theoretical results.

Abstract

We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher network. We analyze the performance of gradient descent for training such kind of neural networks based on empirical risk minimization, and provide algorithm-dependent guarantees. In particular, we prove that tensor initialization followed by gradient descent can converge to the ground-truth parameters at a linear rate up to some statistical error. To the best of our knowledge, this is the first work characterizing the recovery guarantee for practical learning of one-hidden-layer ReLU networks with multiple neurons. Numerical experiments verify our theoretical findings.