Saddle Point Evasion via Curvature-Regularized Gradient Dynamics

TL;DR

This paper introduces Curvature-Regularized Gradient Dynamics (CRGD), a novel method that effectively escapes saddle points in nonconvex optimization by penalizing negative Hessian eigenvalues, ensuring convergence to second-order stationary points.

Contribution

The paper proposes CRGD, a new curvature-regularized gradient method that guarantees convergence to second-order stationary points in nonconvex optimization.

Findings

CRGD converges to second-order stationary points.

CRGD outperforms gradient descent in saddle point escape.

Numerical experiments validate CRGD's effectiveness.

Abstract

Nonconvex optimization underlies many modern machine learning and control tasks, where saddle points pose the dominant obstacle to reliable convergence in high-dimensional settings. Escaping these saddle points deterministically using continuous-time optimization remains an open challenge: gradient descent is blind to curvature, stochastic perturbation methods lack deterministic guarantees, and Newton-type approaches suffer from Hessian singularity. Adopting the perspective of viewing optimization algorithms as dynamical systems, we present Curvature-Regularized Gradient Dynamics (CRGD), which augments the objective with a smooth penalty on the negative Hessian eigenvalues, yielding an augmented cost that serves as an optimization Lyapunov function with user-selectable convergence rates to second-order stationary points. Numerical experiments confirm that CRGD converges to second-order stationary points, even in regimes where gradient descent fails.