Small nonlinearities in activation functions create bad local minima in neural networks

TL;DR

This paper demonstrates that even small nonlinearities in neural network activation functions can lead to numerous bad local minima, challenging the assumption that such issues are limited to deep linear networks.

Contribution

It proves the existence of spurious local minima in shallow nonlinear networks for most datasets, highlighting limitations of linear network insights.

Findings

Existence of many local minima in ReLU-like networks

Counterexamples for common nonlinear activations

Unified characterization of global optima in deep linear networks

Abstract

We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima. We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum. Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks. We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.