Learning to Recover for Safe Reinforcement Learning

TL;DR

This paper introduces TU-Recovery, a three-stage architecture for safe reinforcement learning that learns safety controllers through algorithms, improving safety and efficiency in complex environments.

Contribution

It proposes a novel three-stage architecture for safe RL, including a safety critic, recovery policy, and auxiliary reward to mitigate adversarial phenomena.

Findings

TU-Recovery outperforms unconstrained methods in reward and safety.

Auxiliary reward significantly reduces constraint violations.

The approach is validated in robot navigation experiments.

Abstract

Safety controllers is widely used to achieve safe reinforcement learning. Most methods that apply a safety controller are using handcrafted safety constraints to construct the safety controller. However, when the environment dynamics are sophisticated, handcrafted safety constraints become unavailable. Therefore, it worth to research on constructing safety controllers by learning algorithms. We propose a three-stage architecture for safe reinforcement learning, namely TU-Recovery Architecture. A safety critic and a recovery policy is learned before task training. They form a safety controller to ensure safety in task training. Then a phenomenon induced by disagreement between task policy and recovery policy, called adversarial phenomenon, which reduces learning efficiency and model performance, is described. Auxiliary reward is proposed to mitigate adversarial phenomenon, while help the task policy to learn to recover from high-risk states. A series of experiments are conducted in a robot navigation environment. Experiments demonstrate that TU-Recovery outperforms unconstrained counterpart in both reward gaining and constraint violations during task training, and auxiliary reward further improve TU-Recovery in reward-to-cost ratio by significantly reduce constraint violations.