FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimize

TL;DR

This paper introduces FISAR, a novel deep neural network-based optimizer for safe reinforcement learning that guarantees forward invariance of safety constraints, ensuring constraint violations decrease monotonically in safety-critical environments.

Contribution

It proposes the first DNN-based optimizer for constrained optimization with forward invariance guarantees, addressing the limitations of classic algorithms in safety-critical RL tasks.

Findings

The optimizer effectively reduces constraint violations in experiments.

It maximizes cumulative reward while maintaining safety constraints.

Validated on numerical optimization and navigation tasks.

Abstract

This paper investigates reinforcement learning with constraints, which are indispensable in safety-critical environments. To drive the constraint violation monotonically decrease, we take the constraints as Lyapunov functions and impose new linear constraints on the policy parameters' updating dynamics. As a result, the original safety set can be forward-invariant. However, because the new guaranteed-feasible constraints are imposed on the updating dynamics instead of the original policy parameters, classic optimization algorithms are no longer applicable. To address this, we propose to learn a generic deep neural network (DNN)-based optimizer to optimize the objective while satisfying the linear constraints. The constraint-satisfaction is achieved via projection onto a polytope formulated by multiple linear inequality constraints, which can be solved analytically with our newly designed metric. To the best of our knowledge, this is the \textit{first} DNN-based optimizer for constrained optimization with the forward invariance guarantee. We show that our optimizer trains a policy to decrease the constraint violation and maximize the cumulative reward monotonically. Results on numerical constrained optimization and obstacle-avoidance navigation validate the theoretical findings.