A Generalized Energy-Based Adaptive Gradient Method for Optimization

TL;DR

This paper introduces a generalized energy-based adaptive gradient method that enhances stability and convergence in optimization, demonstrating superior performance over existing methods through theoretical analysis and numerical experiments.

Contribution

It extends AEGD by generalizing the energy function, maintaining unconditional stability, and establishing optimal convergence rates with improved practical performance.

Findings

gAEGD preserves energy stability and robustness to step size.

The method achieves an $O(1/k)$ convergence rate for stationary points.

Numerical results show gAEGD outperforms AEGD on benchmarks.

Abstract

Adaptive Gradient Descent with Energy (AEGD) is a variant of gradient descent (GD) designed to mitigate step-size sensitivity through an energy-based formulation. AEGD is notable for its unconditional energy stability, which guarantees monotonic energy decay and convergence independently of the initial step size. In this work, we propose the Generalized Energy-Based Adaptive Gradient (gAEGD) method, which extends AEGD by replacing the square-root energy with a broader class of admissible energy functions. We show that gAEGD preserves unconditional energy stability, remains robust to step-size selection, and exhibits a two-phase adaptive dynamic: the effective step size first adjusts automatically and then stabilizes within a regime that ensures decay of the objective function values. We establish an optimal convergence rate of $O(\frac{1}{k})$ for attaining an $\epsilon$-stationary point, and further derive improved convergence rates for the objective gap under a local Kurdyka-{\L}ojasiewicz (KL) condition. Extensive numerical experiments corroborate our theoretical results and demonstrate that a particular instance of gAEGD-ALEGD, which employs a logarithmic energy function-often outperforms AEGD across a range of benchmark optimization problems.