Global optimality conditions for deep neural networks

TL;DR

This paper establishes necessary and sufficient conditions for global optimality in deep linear and nonlinear neural networks, providing practical tests for global minima in nonconvex loss landscapes.

Contribution

It introduces efficiently checkable conditions for global optimality in deep linear networks and extends these results to nonlinear networks within a limited setting.

Findings

Necessary and sufficient conditions for deep linear networks' global minima

Efficient test for global optimality in nonconvex loss landscapes

Extension of conditions to deep nonlinear networks

Abstract

We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.