Feasibility Consistent Representation Learning for Safe Reinforcement Learning

TL;DR

This paper introduces FCSRL, a novel framework that combines representation learning with safety constraints to improve safe reinforcement learning, especially in estimating safety metrics from raw states.

Contribution

The paper proposes a new framework that integrates self-supervised representation learning with safety constraints to enhance safe RL performance and safety estimation accuracy.

Findings

Outperforms previous baselines in safety-aware embedding learning

Achieves better safety constraint estimation from raw states

Demonstrates effectiveness on vector and image-based tasks

Abstract

In the field of safe reinforcement learning (RL), finding a balance between satisfying safety constraints and optimizing reward performance presents a significant challenge. A key obstacle in this endeavor is the estimation of safety constraints, which is typically more difficult than estimating a reward metric due to the sparse nature of the constraint signals. To address this issue, we introduce a novel framework named Feasibility Consistent Safe Reinforcement Learning (FCSRL). This framework combines representation learning with feasibility-oriented objectives to identify and extract safety-related information from the raw state for safe RL. Leveraging self-supervised learning techniques and a more learnable safety metric, our approach enhances the policy learning and constraint estimation. Empirical evaluations across a range of vector-state and image-based tasks demonstrate that our method is capable of learning a better safety-aware embedding and achieving superior performance than previous representation learning baselines.