Adaptive Preconditioned Gradient Descent with Energy

TL;DR

This paper introduces an adaptive energy-based step size method for preconditioned gradient descent algorithms, improving stability and convergence in constrained optimization problems.

Contribution

It integrates preconditioned gradient methods into the Adaptive Energy Gradient Descent framework, providing theoretical analysis and demonstrating superior numerical performance.

Findings

Unconditional energy-stability established for the proposed method

Convergence rates analyzed for various objective functions

Numerical experiments show excellent optimization performance

Abstract

We propose an adaptive step size with an energy approach for a suitable class of preconditioned gradient descent methods. We focus on settings where the preconditioning is applied to address the constraints in optimization problems, such as the Hessian-Riemannian and natural gradient descent methods. More specifically, we incorporate these preconditioned gradient descent algorithms in the recently introduced Adaptive Energy Gradient Descent (AEGD) framework. In particular, we discuss theoretical results on the unconditional energy-stability and convergence rates across three classes of objective functions. Furthermore, our numerical results demonstrate excellent performance of the proposed method on several test bed optimization problems.