The Power of Normalization: Faster Evasion of Saddle Points

TL;DR

This paper demonstrates that normalized gradient descent, with proper parameters and noise, can efficiently escape saddle points and converge to local minima faster than existing methods, with applications to tensor decomposition.

Contribution

The paper provides a theoretical analysis showing NGD's ability to evade saddle points and improve convergence rates, supported by practical application to online tensor decomposition.

Findings

NGD can provably escape saddle points with appropriate parameters.

NGD achieves faster convergence rates than previous first-order algorithms.

Application to tensor decomposition shows practical effectiveness.

Abstract

A commonly used heuristic in non-convex optimization is Normalized Gradient Descent (NGD) - a variant of gradient descent in which only the direction of the gradient is taken into account and its magnitude ignored. We analyze this heuristic and show that with carefully chosen parameters and noise injection, this method can provably evade saddle points. We establish the convergence of NGD to a local minimum, and demonstrate rates which improve upon the fastest known first order algorithm due to Ge e al. (2015). The effectiveness of our method is demonstrated via an application to the problem of online tensor decomposition; a task for which saddle point evasion is known to result in convergence to global minima.