Safe Reinforcement Learning with Contrastive Risk Prediction

TL;DR

This paper introduces a risk preventive training method for safe reinforcement learning that uses a contrastive classifier to predict unsafe states, reshaping rewards to promote safety in robotic simulations.

Contribution

It presents a novel risk prediction approach using a contrastive classifier and integrates it into safe RL to improve safety without sacrificing performance.

Findings

Comparable performance to state-of-the-art model-based methods

Outperforms traditional model-free safe RL approaches

Effective in robotic simulation environments

Abstract

As safety violations can lead to severe consequences in real-world robotic applications, the increasing deployment of Reinforcement Learning (RL) in robotic domains has propelled the study of safe exploration for reinforcement learning (safe RL). In this work, we propose a risk preventive training method for safe RL, which learns a statistical contrastive classifier to predict the probability of a state-action pair leading to unsafe states. Based on the predicted risk probabilities, we can collect risk preventive trajectories and reshape the reward function with risk penalties to induce safe RL policies. We conduct experiments in robotic simulation environments. The results show the proposed approach has comparable performance with the state-of-the-art model-based methods and outperforms conventional model-free safe RL approaches.