Robust Hybrid Zero-Order Optimization Algorithms with Acceleration via Averaging in Time

TL;DR

This paper introduces robust zero-order optimization algorithms with acceleration via averaging, applicable to various convex problems, and provides a new averaging theorem for hybrid dynamical systems.

Contribution

The paper develops a new class of accelerated zero-order algorithms modeled as hybrid systems and introduces a novel averaging theorem applicable to these systems.

Findings

Established robust stability for the proposed algorithms.

Derived a new averaging theorem for hybrid systems.

Validated results with numerical examples.

Abstract

We study novel robust zero-order algorithms with acceleration for the solution of real-time optimization problems. In particular, we propose a family of extremum seeking dynamics that can be universally modeled as singularly perturbed hybrid dynamical systems with restarting mechanisms. From this family of dynamics, we synthesize four fast algorithms for the solution of convex, strongly convex, constrained, and unconstrained optimization problems. In each case, we establish robust semi-global practical asymptotic or exponential stability results, and we show how to obtain well-posed discretized algorithms that retain the main properties of the original dynamics. Given that existing averaging theorems for singularly perturbed hybrid systems are not directly applicable to our setting, we derive a new averaging theorem that relaxes some of the assumptions made in the literature, allowing us to make a clear link between the KL bounds that characterize the rates of convergence of the hybrid dynamics and their average dynamics. We also show that our results are applicable to non-hybrid algorithms, thus providing a general framework for accelerated dynamics based on averaging theory. We present different numerical examples to illustrate our results.