Accelerated Continuous-Time Approximate Dynamic Programming via Data-Assisted Hybrid Control

TL;DR

This paper presents a novel continuous-time control algorithm that accelerates convergence using dynamic momentum and data-assisted techniques, eliminating the need for persistent excitation and ensuring stability.

Contribution

It introduces the first momentum-based actor-critic algorithm for continuous-time systems that leverages past data and hybrid resets to improve convergence and stability.

Findings

Accelerates convergence of continuous-time control systems.

Eliminates the need for persistent excitation conditions.

Ensures asymptotic stability using hybrid dynamical systems theory.

Abstract

We introduce a new closed-loop architecture for the online solution of approximate optimal control problems in the context of continuous-time systems. Specifically, we introduce the first algorithm that incorporates dynamic momentum in actor-critic structures to control continuous-time dynamic plants with an affine structure in the input. By incorporating dynamic momentum in our algorithm, we are able to accelerate the convergence properties of the closed-loop system, achieving superior transient performance compared to traditional gradient-descent based techniques. In addition, by leveraging the existence of past recorded data with sufficiently rich information properties, we dispense with the persistence of excitation condition traditionally imposed on the regressors of the critic and the actor. Given that our continuous-time momentum-based dynamics also incorporate periodic discrete-time resets that emulate restarting techniques used in the machine learning literature, we leverage tools from hybrid dynamical systems theory to establish asymptotic stability properties for the closed-loop system. We illustrate our results with a numerical example.