Convergence rates of regularized quasi-Newton methods without strong convexity

TL;DR

This paper establishes non-asymptotic super-linear convergence rates for regularized quasi-Newton methods applied to non-smooth, non-convex problems satisfying the Kurdyka-ojasiewicz property, without relying on strong convexity or line search.

Contribution

It provides the first explicit non-asymptotic convergence rates for regularized quasi-Newton methods on non-convex, non-smooth problems with K\u00e1lka-ojasiewicz property, independent of strong convexity.

Findings

Super-linear convergence rates are established without line search or trust region.

Rates are explicit and non-asymptotic for non-convex, non-smooth problems.

Convergence results are extended to convex problems using gradient regularized quasi-Newton methods.

Abstract

In this paper, we study convergence rates of the cubic regularized proximal quasi-Newton method (\csr) for solving non-smooth additive composite problems that satisfy the so-called Kurdyka-\L ojasiewicz (K\L ) property with respect to some desingularization function $\phi$ rather than strong convexity. After a number of iterations $k_0$, Cubic SR1 PQN exhibits non-asymptotic explicit super-linear convergence rates for any $k\geq k_0$. In particular, when $\phi(t)=ct^{1/2}$, Cubic SR1 PQN has a convergence rate of order $\left(\frac{C}{(k-k_0)^{1/2}}\right)^{(k-k_0)/2}$, where $k$ is the number of iterations and $C>0$ is a constant. For the special case, i.e. functions which satisfy \L ojasiewicz inequality, the rate becomes global and non-asymptotic. This work presents, for the first time, non-asymptotic explicit convergence rates of regularized (proximal) SR1 quasi-Newton methods applied to non-convex non-smooth problems with K\L\ property. Actually, the rates are novel even in the smooth non-convex case. Notably, we achieve this without employing line search or trust region strategies, without assuming the Dennis-Mor\'e condition, without any assumptions on quasi-Newton metrics and without assuming strong convexity. Furthermore, for convex problems, we focus on a more tractable gradient regularized quasi-Newton method (Grad SR1 PQN) which can achieve results similar to those obtained with cubic regularization. We also demonstrate, for the first time, the non-asymptotic super-linear convergence rate of Grad SR1 PQN for solving convex problems with the help of the \L ojasiewicz inequality instead of strong convexity.