Sufficient conditions for instability of the subgradient method with constant step size

TL;DR

This paper identifies conditions under which the subgradient method becomes unstable with a constant step size near local minima, especially in complex models like neural networks and PCA.

Contribution

It provides new sufficient conditions for instability of the subgradient method around local minima in semi-algebraic functions, applicable to neural networks and PCA.

Findings

Conditions satisfied by spurious minima in neural networks

Instability occurs near certain local minima in semi-algebraic functions

Applicable to robust PCA and neural network training scenarios

Abstract

We provide sufficient conditions for instability of the subgradient method with constant step size around a local minimum of a locally Lipschitz semi-algebraic function. They are satisfied by several spurious local minima arising in robust principal component analysis and neural networks.