Don't do it: Safer Reinforcement Learning With Rule-based Guidance

TL;DR

This paper introduces a rule-based safety mechanism for reinforcement learning that overrides unsafe actions, leading to safer training, faster convergence, and improved performance.

Contribution

It proposes a safe epsilon-greedy algorithm that incorporates safety rules to prevent harmful actions during reinforcement learning.

Findings

Significantly increased safety during training.

Faster convergence compared to baseline.

Achieved better overall performance.

Abstract

During training, reinforcement learning systems interact with the world without considering the safety of their actions. When deployed into the real world, such systems can be dangerous and cause harm to their surroundings. Often, dangerous situations can be mitigated by defining a set of rules that the system should not violate under any conditions. For example, in robot navigation, one safety rule would be to avoid colliding with surrounding objects and people. In this work, we define safety rules in terms of the relationships between the agent and objects and use them to prevent reinforcement learning systems from performing potentially harmful actions. We propose a new safe epsilon-greedy algorithm that uses safety rules to override agents' actions if they are considered to be unsafe. In our experiments, we show that a safe epsilon-greedy policy significantly increases the safety of the agent during training, improves the learning efficiency resulting in much faster convergence, and achieves better performance than the base model.