Continuous-Time Accelerated Methods via a Hybrid Control Lens

TL;DR

This paper introduces two novel classes of accelerated optimization methods modeled as hybrid control systems, achieving exponential convergence rates under the Polyak--Łojasiewicz condition, with guarantees of Zeno-free trajectories.

Contribution

It proposes hybrid control-based accelerated methods that dynamically synthesize feedback controls, offering a new perspective beyond parametric differential equations for optimization.

Findings

Achieves exponential convergence under Polyak--Łojasiewicz inequality.

Ensures Zeno-free solution trajectories in hybrid control systems.

Provides a discretization mechanism for exponential convergence.

Abstract

Treating optimization methods as dynamical systems can be traced back centuries ago in order to comprehend the notions and behaviors of optimization methods. Lately, this mind set has become the driving force to design new optimization methods. Inspired by the recent dynamical system viewpoint of Nesterov's fast method, we propose two classes of fast methods, formulated as hybrid control systems, to obtain pre-specified exponential convergence rate. Alternative to the existing fast methods which are parametric-in-time second order differential equations, we dynamically synthesize feedback controls in a state-dependent manner. Namely, in the first class the damping term is viewed as the control input, while in the second class the amplitude with which the gradient of the objective function impacts the dynamics serves as the controller. The objective function requires to satisfy the so-called Polyak--{\L}ojasiewicz inequality which effectively implies no local optima and a certain gradient-domination property. Moreover, we establish that both hybrid structures possess Zeno-free solution trajectories. We finally provide a mechanism to determine the discretization step size to attain an exponential convergence rate.