Heavy Ball and Nesterov Accelerations with Hessian-driven Damping for Nonconvex Optimization

TL;DR

This paper introduces new Hessian-driven damping methods for nonconvex optimization, deriving discrete algorithms with improved stability and convergence, supported by theoretical analysis and numerical experiments.

Contribution

It develops novel Heavy Ball and Nesterov-type algorithms with Hessian correction for strongly quasiconvex functions, bridging continuous dynamics and discrete optimization.

Findings

Linear convergence of the proposed algorithms.

Enhanced stability and reduced oscillations.

Numerical experiments confirm theoretical results.

Abstract

In this work, we investigate a second-order dynamical system with Hessian-driven damping tailored for a class of nonconvex functions called strongly quasiconvex. Buil\-ding upon this continuous-time model, we derive two discrete-time gra\-dient-based algorithms through time discretizations. The first is a Heavy Ball method with Hessian correction, incorporating cur\-va\-tu\-re-dependent terms that arise from discretizing the Hessian damping component. The second is a Nesterov-type accelerated method with adaptive momentum, fea\-tu\-ring correction terms that account for local curvature. Both algorithms aim to enhance stability and convergence performance, particularly by mi\-ti\-ga\-ting oscillations commonly observed in cla\-ssi\-cal momentum me\-thods. Furthermore, in both cases we establish li\-near convergence to the optimal solution for the iterates and functions values. Our approach highlights the rich interplay between continuous-time dynamics and discrete optimization algorithms in the se\-tting of strongly quasiconvex objectives. Numerical experiments are presented to support obtained results.