Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization

TL;DR

This paper provides a theoretical explanation for why gradient descent effectively trains large-scale neural networks by showing their optimization landscape is convex in a transformed space, leading to guaranteed convergence.

Contribution

The authors introduce the canonical space and disparity matrix to prove that neural network training behaves like convex optimization under certain conditions.

Findings

Objective functions are convex in the canonical space.

Gradient descent converges to global minima if disparity matrices are full rank.

In over-parameterized NNs, convergence to zero loss is highly probable.

Abstract

In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning NNs are convex in the canonical model space. We further elucidate that the gradients between the original NN model space and the canonical space are related by a pointwise linear transformation, which is represented by the so-called disparity matrix. Furthermore, we have proved that gradient descent methods surely converge to a global minimum of zero loss provided that the disparity matrices maintain full rank. If this full-rank condition holds, the learning of NNs behaves in the same way as normal convex optimization. At last, we have shown that the chance to have singular disparity matrices is extremely slim in large NNs. In particular, when over-parameterized NNs are randomly initialized, the gradient decent algorithms converge to a global minimum of zero loss in probability.