A Homogeneous Second-Order Descent Method for Nonconvex Optimization

TL;DR

This paper presents a new second-order descent method for nonconvex optimization that efficiently finds approximate stationary points with proven convergence rates and demonstrated numerical advantages.

Contribution

Introduction of HSODM, a single-loop second-order method using homogenized quadratic approximation with proven global and local convergence properties.

Findings

Achieves $O(\epsilon^{-3/2})$ global convergence rate.

Exhibits local quadratic convergence under standard assumptions.

Numerical results show advantages over existing second-order methods.

Abstract

In this paper, we introduce a Homogeneous Second-Order Descent Method (HSODM) using the homogenized quadratic approximation to the original function. The merit of homogenization is that only the leftmost eigenvector of a gradient-Hessian integrated matrix is computed at each iteration. Therefore, the algorithm is a single-loop method that does not need to switch to other sophisticated algorithms and is easy to implement. We show that HSODM has a global convergence rate of $O(\epsilon^{-3/2})$ to find an $\epsilon$-approximate second-order stationary point, and has a local quadratic convergence rate under the standard assumptions. The numerical results demonstrate the advantage of the proposed method over other second-order methods.