Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima

TL;DR

This paper demonstrates that gradient descent with weight normalization can successfully learn a one-hidden-layer CNN despite the existence of spurious local minima, revealing nuanced dynamics and convergence properties.

Contribution

It proves that gradient descent can recover true parameters in a CNN with spurious minima, and analyzes the two-phase convergence behavior.

Findings

Gradient descent can recover true parameters despite spurious local minima.

The convergence has two phases: slow initial progress, then rapid convergence.

Multiple restarts boost the probability of successful learning.

Abstract

We consider the problem of learning a one-hidden-layer neural network with non-overlapping convolutional layer and ReLU activation, i.e., $f(\mathbf{Z}, \mathbf{w}, \mathbf{a}) = \sum_j a_j\sigma(\mathbf{w}^T\mathbf{Z}_j)$, in which both the convolutional weights $\mathbf{w}$ and the output weights $\mathbf{a}$ are parameters to be learned. When the labels are the outputs from a teacher network of the same architecture with fixed weights $(\mathbf{w}^*, \mathbf{a}^*)$, we prove that with Gaussian input $\mathbf{Z}$, there is a spurious local minimizer. Surprisingly, in the presence of the spurious local minimizer, gradient descent with weight normalization from randomly initialized weights can still be proven to recover the true parameters with constant probability, which can be boosted to probability $1$ with multiple restarts. We also show that with constant probability, the same procedure could also converge to the spurious local minimum, showing that the local minimum plays a non-trivial role in the dynamics of gradient descent. Furthermore, a quantitative analysis shows that the gradient descent dynamics has two phases: it starts off slow, but converges much faster after several iterations.