A reinforcement learning method with closed-loop stability guarantee

TL;DR

This paper introduces a reinforcement learning method that guarantees semi-global stability of control systems by integrating a control Lyapunov function into the RL scheme, ensuring system stability during learning.

Contribution

It develops an online RL scheme that guarantees practical stability using a control Lyapunov function within a Lyapunov-like constraint framework.

Findings

Ensures closed-loop stability with RL in control systems.

Optimizes cost function while maintaining stability.

Validated on a non-holonomic integrator case study.

Abstract

Reinforcement learning (RL) in the context of control systems offers wide possibilities of controller adaptation. Given an infinite-horizon cost function, the so-called critic of RL approximates it with a neural net and sends this information to the controller (called "actor"). However, the issue of closed-loop stability under an RL-method is still not fully addressed. Since the critic delivers merely an approximation to the value function of the corresponding infinite-horizon problem, no guarantee can be given in general as to whether the actor's actions stabilize the system. Different approaches to this issue exist. The current work offers a particular one, which, starting with a (not necessarily smooth) control Lyapunov function (CLF), derives an online RL-scheme in such a way that practical semi-global stability property of the closed-loop can be established. The approach logically continues the work of the authors on parameterized controllers and Lyapunov-like constraints for RL, whereas the CLF now appears merely in one of the constraints of the control scheme. The analysis of the closed-loop behavior is done in a sample-and-hold (SH) manner thus offering a certain insight into the digital realization. The case study with a non-holonomic integrator shows the capabilities of the derived method to optimize the given cost function compared to a nominal stabilizing controller.