The Global Optimization Geometry of Shallow Linear Neural Networks

TL;DR

This paper analyzes the loss landscape of shallow linear neural networks, showing it has no spurious local minima and saddle points with negative curvature, enabling provable global convergence of common algorithms.

Contribution

It establishes that shallow linear neural networks have benign geometric properties with milder assumptions than prior work, ensuring global convergence of local search algorithms.

Findings

No spurious local minima in the loss landscape

Saddle points have negative curvature directions

Gradient descent can provably find global minima

Abstract

We examine the squared error loss landscape of shallow linear neural networks. We show---with significantly milder assumptions than previous works---that the corresponding optimization problems have benign geometric properties: there are no spurious local minima and the Hessian at every saddle point has at least one negative eigenvalue. This means that at every saddle point there is a directional negative curvature which algorithms can utilize to further decrease the objective value. These geometric properties imply that many local search algorithms (such as the gradient descent which is widely utilized for training neural networks) can provably solve the training problem with global convergence.