Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition

TL;DR

This paper proves that stochastic gradient descent can efficiently escape saddle points and find local minima in non-convex tensor decomposition problems, providing the first global convergence guarantees for such settings.

Contribution

It introduces the strict saddle property for non-convex functions and applies it to develop an online algorithm with global convergence guarantees for tensor decomposition.

Findings

SGD converges to local minima in polynomial time

First global convergence guarantee for non-convex functions with many local minima

New formulation for tensor decomposition with strict saddle property

Abstract

We analyze stochastic gradient descent for optimizing non-convex functions. In many cases for non-convex functions the goal is to find a reasonable local minimum, and the main concern is that gradient updates are trapped in saddle points. In this paper we identify strict saddle property for non-convex problem that allows for efficient optimization. Using this property we show that stochastic gradient descent converges to a local minimum in a polynomial number of iterations. To the best of our knowledge this is the first work that gives global convergence guarantees for stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points. Our analysis can be applied to orthogonal tensor decomposition, which is widely used in learning a rich class of latent variable models. We propose a new optimization formulation for the tensor decomposition problem that has strict saddle property. As a result we get the first online algorithm for orthogonal tensor decomposition with global convergence guarantee.