Homogeneous Second-Order Descent Framework: A Fast Alternative to Newton-Type Methods

TL;DR

This paper introduces HSODF, a second-order descent framework that offers a faster alternative to Newton methods for optimization, especially effective in ill-conditioned and high-dimensional problems.

Contribution

The paper develops a novel homogeneous second-order descent framework that generalizes existing models, enabling adaptive and efficient optimization methods with proven complexity bounds.

Findings

Adaptive HSODM achieves $O( ext{ extphi})^{-3/2}$ complexity for nonconvex problems.

Homotopy HSODM attains global linear convergence without strong convexity.

Preliminary numerical results demonstrate effectiveness on ill-conditioned, high-dimensional problems.

Abstract

This paper proposes a homogeneous second-order descent framework (HSODF) for nonconvex and convex optimization based on the generalized homogeneous model (GHM). In comparison to the Newton steps, the GHM can be solved by extremal symmetric eigenvalue procedures and thus grant an advantage in ill-conditioned problems. Moreover, GHM extends the ordinary homogeneous model (OHM) (Zhang et al. 2022) to allow adaptiveness in the construction of the aggregated matrix. Consequently, HSODF is able to recover some well-known second-order methods, such as trust-region methods and gradient regularized methods, while maintaining comparable iteration complexity bounds. We also study two specific realizations of HSODF. One is adaptive HSODM, which has a parameter-free $O(\epsilon^{-3/2})$ global complexity bound for nonconvex second-order Lipschitz continuous objective functions. The other one is homotopy HSODM, which is proven to have a global linear rate of convergence without strong convexity. The efficiency of our approach to ill-conditioned and high-dimensional problems is justified by some preliminary numerical results.