Depth Creates No Bad Local Minima

TL;DR

This paper proves that depth alone, without nonlinearity, does not create bad local minima in deep linear neural networks, simplifying previous proofs and extending understanding of loss surface properties.

Contribution

It demonstrates that depth alone does not cause bad local minima in linear networks and simplifies existing proofs, broadening theoretical insights into deep linear models.

Findings

All local minima in deep linear networks are global minima.

Depth alone does not create bad local minima without nonlinearity.

The analysis extends to loss functions beyond square loss.

Abstract

In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep linear neural networks are global minima. Our theoretical results generalize previous results with fewer assumptions, and this analysis provides a method to show similar results beyond square loss in deep linear models.