Improving Safety in Deep Reinforcement Learning using Unsupervised Action Planning

TL;DR

This paper introduces an unsupervised action planning method to enhance safety in deep reinforcement learning, using a safety buffer and clustering to recover from dangerous states, leading to higher rewards in robotic tasks.

Contribution

It presents a novel safety mechanism for deep RL that employs unsupervised learning to store and select recovery actions, improving safety without supervision.

Findings

Achieves higher rewards than baselines in robotic control tasks.

Effective safety recovery in both navigation and manipulation tasks.

Utilizes unsupervised clustering for safety action selection.

Abstract

One of the key challenges to deep reinforcement learning (deep RL) is to ensure safety at both training and testing phases. In this work, we propose a novel technique of unsupervised action planning to improve the safety of on-policy reinforcement learning algorithms, such as trust region policy optimization (TRPO) or proximal policy optimization (PPO). We design our safety-aware reinforcement learning by storing all the history of "recovery" actions that rescue the agent from dangerous situations into a separate "safety" buffer and finding the best recovery action when the agent encounters similar states. Because this functionality requires the algorithm to query similar states, we implement the proposed safety mechanism using an unsupervised learning algorithm, k-means clustering. We evaluate the proposed algorithm on six robotic control tasks that cover navigation and manipulation. Our results show that the proposed safety RL algorithm can achieve higher rewards compared with multiple baselines in both discrete and continuous control problems. The supplemental video can be found at: https://youtu.be/AFTeWSohILo.