Qualitatively characterizing neural network optimization problems

TL;DR

This paper introduces a simple analysis method revealing that modern neural networks typically do not face significant obstacles like local minima during training, challenging previous beliefs about their optimization difficulty.

Contribution

The study provides evidence that state-of-the-art neural networks can be trained effectively without encountering major local optima, using a straightforward analysis along the training path.

Findings

Neural networks often do not encounter significant obstacles during training.

Modern neural networks can achieve negligible training error with direct stochastic gradient descent.

The analysis technique offers new insights into neural network optimization landscapes.

Abstract

Training neural networks involves solving large-scale non-convex optimization problems. This task has long been believed to be extremely difficult, with fear of local minima and other obstacles motivating a variety of schemes to improve optimization, such as unsupervised pretraining. However, modern neural networks are able to achieve negligible training error on complex tasks, using only direct training with stochastic gradient descent. We introduce a simple analysis technique to look for evidence that such networks are overcoming local optima. We find that, in fact, on a straight path from initialization to solution, a variety of state of the art neural networks never encounter any significant obstacles.