Convergence of Cubic Regularization for Nonconvex Optimization under KL Property

TL;DR

This paper analyzes the asymptotic convergence rates of cubic-regularized Newton's method for nonconvex optimization by leveraging the KL property, revealing faster convergence than gradient descent across various optimality measures.

Contribution

It characterizes the asymptotic convergence rates of CR under the KL property for multiple optimality measures, broadening understanding beyond specific geometric conditions.

Findings

CR converges faster than gradient descent under KL property

Convergence rates are characterized for function value, variable distance, gradient norm, and Hessian eigenvalues

Results apply across the full parameter regime of the KL property

Abstract

Cubic-regularized Newton's method (CR) is a popular algorithm that guarantees to produce a second-order stationary solution for solving nonconvex optimization problems. However, existing understandings of the convergence rate of CR are conditioned on special types of geometrical properties of the objective function. In this paper, we explore the asymptotic convergence rate of CR by exploiting the ubiquitous Kurdyka-Lojasiewicz (KL) property of nonconvex objective functions. In specific, we characterize the asymptotic convergence rate of various types of optimality measures for CR including function value gap, variable distance gap, gradient norm and least eigenvalue of the Hessian matrix. Our results fully characterize the diverse convergence behaviors of these optimality measures in the full parameter regime of the KL property. Moreover, we show that the obtained asymptotic convergence rates of CR are order-wise faster than those of first-order gradient descent algorithms under the KL property.