Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction

TL;DR

This paper introduces ASCPO, a new reinforcement learning algorithm that guarantees high-probability satisfaction of state-wise safety constraints in stochastic systems, outperforming existing methods in robot control tasks.

Contribution

We propose ASCPO, a policy optimization algorithm that enforces state-wise safety with high probability without strong assumptions, advancing safe RL in real-world applications.

Findings

ASCPO achieves high-probability safety constraint satisfaction in continuous control tasks.

ASCPO outperforms existing safe RL methods in robot locomotion experiments.

The approach is effective for complex, real-world-like safety constraints.

Abstract

Enforcing state-wise safety constraints is critical for the application of reinforcement learning (RL) in real-world problems, such as autonomous driving and robot manipulation. However, existing safe RL methods only enforce state-wise constraints in expectation or enforce hard state-wise constraints with strong assumptions. The former does not exclude the probability of safety violations, while the latter is impractical. Our insight is that although it is intractable to guarantee hard state-wise constraints in a model-free setting, we can enforce state-wise safety with high probability while excluding strong assumptions. To accomplish the goal, we propose Absolute State-wise Constrained Policy Optimization (ASCPO), a novel general-purpose policy search algorithm that guarantees high-probability state-wise constraint satisfaction for stochastic systems. We demonstrate the effectiveness of our approach by training neural network policies for extensive robot locomotion tasks, where the agent must adhere to various state-wise safety constraints. Our results show that ASCPO significantly outperforms existing methods in handling state-wise constraints across challenging continuous control tasks, highlighting its potential for real-world applications.