Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller

TL;DR

This paper introduces a safe reinforcement learning control method for continuous-time nonlinear systems that guarantees safety without needing a backup controller, using barrier Lyapunov functions and an Actor-Critic approach.

Contribution

It presents a novel on-policy RL algorithm that enforces safety constraints via BLFs without external backup controllers for uncertain nonlinear systems.

Findings

Successfully applied to Euler-Lagrange systems

Ensures safety throughout the learning process

No need for external backup controllers

Abstract

This paper proposes an on-policy reinforcement learning (RL) control algorithm that solves the optimal regulation problem for a class of uncertain continuous-time nonlinear systems under user-defined state constraints. We formulate the safe RL problem as the minimization of the Hamiltonian subject to a constraint on the time-derivative of a barrier Lyapunov function (BLF). We subsequently use the analytical solution of the optimization problem to modify the Actor-Critic-Identifier architecture to learn the optimal control policy safely. The proposed method does not require the presence of external backup controllers, and the RL policy ensures safety for the entire duration. The efficacy of the proposed controller is demonstrated on a class of Euler-Lagrange systems.