Whiplash Gradient Descent Dynamics

TL;DR

This paper introduces the Whiplash Inertial Gradient dynamics, a novel optimization method with convergence guarantees and heuristics for escaping saddle points, demonstrating promising convergence rates for quadratic functions.

Contribution

The paper presents the Whiplash Inertial Gradient dynamics, including its convergence analysis, relaxation sequences, and a heuristic for saddle point escape, advancing optimization algorithms.

Findings

Polynomial and exponential convergence rates for quadratic costs

Introduction of symplectic asymptotic convergence analysis

Development of a heuristic to escape saddle points

Abstract

In this paper, we propose the Whiplash Inertial Gradient dynamics, a closed-loop optimization method that utilises gradient information, to find the minima of a cost function in finite-dimensional settings. We introduce the symplectic asymptotic convergence analysis for the Whiplash system for convex functions. We also introduce relaxation sequences to explain the non-classical nature of the algorithm and an exploring heuristic variant of the Whiplash algorithm to escape saddle points, deterministically. We study the algorithm's performance for various costs and provide a practical methodology for analyzing convergence rates using integral constraint bounds and a novel Lyapunov rate method. Our results demonstrate polynomial and exponential rates of convergence for quadratic cost functions.