Depth with Nonlinearity Creates No Bad Local Minima in ResNets

TL;DR

This paper proves that deep residual networks with nonlinear activations have no bad local minima, ensuring all local minima are as good as the global minimum, which advances understanding of ResNet optimization.

Contribution

It provides a theoretical proof that depth with nonlinearity in ResNets prevents bad local minima, answering an open question from 2018.

Findings

All local minima are no worse than the global minimum.

Residual representations can further improve the minima.

The result applies to arbitrarily deep ResNets with arbitrary nonlinearities.

Abstract

In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations. As a result, this paper provides an affirmative answer to an open question stated in a paper in the conference on Neural Information Processing Systems 2018. This paper advances the optimization theory of deep learning only for ResNets and not for other network architectures.