Lagrangian-based online safe reinforcement learning for state-constrained systems

TL;DR

This paper introduces a Lagrangian-based safe reinforcement learning algorithm for continuous-time uncertain nonlinear systems, ensuring safety constraints are met during online learning through a novel Actor-Critic-Identifier-Lagrangian approach.

Contribution

It develops an online safe RL method that incorporates a state-dependent Lagrange multiplier and guarantees safety and boundedness, advancing safe control in uncertain nonlinear systems.

Findings

The proposed ACIL algorithm learns control policies safely from online data.

Safety and boundedness are theoretically guaranteed by the method.

Simulation results show improved safety performance over existing RL approaches.

Abstract

This paper proposes a safe reinforcement learning (RL) algorithm that approximately solves the state-constrained optimal control problem for continuous-time uncertain nonlinear systems. We formulate the safe RL problem as the minimization of a Lagrangian that includes the cost functional and a user-defined barrier Lyapunov function (BLF) encoding the state constraints. We show that the analytical solution obtained by the application of Karush-Kuhn-Tucker (KKT) conditions contains a state-dependent expression for the Lagrange multiplier, which is a function of uncertain terms in the system dynamics. We argue that a naive estimation of the Lagrange multiplier may lead to safety constraint violations. To obviate this challenge, we propose an Actor-Critic-Identifier-Lagrangian (ACIL) algorithm that learns optimal control policies from online data without compromising safety. We provide safety and boundedness guarantees with the proposed algorithm and compare its performance with existing offline/online RL methods via a simulation study.