Dynamic behavior for a gradient algorithm with energy and momentum

TL;DR

This paper introduces AGEM, a gradient algorithm combining energy and momentum, with a detailed analysis of its dynamic behavior, convergence properties, and linear rate under specific conditions for non-convex optimization.

Contribution

It presents a novel AGEM algorithm and analyzes its dynamic behavior through a high-resolution ODE system, establishing convergence and linear rates under certain conditions.

Findings

Proves global well-posedness of the ODE system.

Shows convergence to critical points.

Establishes linear convergence rate under Polyak-Łojasiewicz condition.

Abstract

This paper investigates a novel gradient algorithm, AGEM, using both energy and momentum, for addressing general non-convex optimization problems. The solution properties of the AGEM algorithm, including aspects such as uniformly boundedness and convergence to critical points, are examined. The dynamic behavior is studied through a comprehensive analysis of a high-resolution ODE system. This ODE system, being nonlinear, is derived by taking the limit of the discrete scheme while preserving the momentum effect through a rescaling of the momentum parameter. The paper emphasizes the global well-posedness of the ODE system and the time-asymptotic convergence of solution trajectories. Furthermore, we establish a linear convergence rate for objective functions that adhere to the Polyak-{\L}ojasiewicz condition.