On the Design of Safe Continual RL Methods for Control of Nonlinear Systems

TL;DR

This paper investigates the challenges of integrating safety constraints into continual reinforcement learning for nonlinear control systems, highlighting limitations of existing methods and proposing a reward-shaping approach to improve safety and task retention.

Contribution

It identifies safety and continual learning issues in existing algorithms and introduces a reward-shaping method to enhance safety preservation in nonlinear, non-stationary systems.

Findings

Online elastic weight consolidation fails to ensure safety in nonlinear systems.

Constrained policy optimization suffers from catastrophic forgetting.

Reward shaping improves safety and task retention in continual RL.

Abstract

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs in the system, typical safe RL algorithms are unable to adapt while retaining past knowledge. Continual reinforcement learning algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system's safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, online elastic weight consolidation, is unable to satisfy safety constraints in non-linear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, non-linear, and non-stationary dynamical systems.