Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration

TL;DR

This paper introduces COX-Q, an off-policy safe reinforcement learning algorithm that combines cost-aware exploration and distributional value estimation to improve safety and efficiency in safety-critical tasks.

Contribution

The paper proposes a novel off-policy safe RL method, COX-Q, integrating cost-bounded exploration and distributional critics for better safety and sample efficiency.

Findings

COX-Q achieves high sample efficiency in safety-critical tasks.

COX-Q maintains competitive safety performance during testing.

The method effectively controls data collection costs.

Abstract

When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offering high sample efficiency, suffer from constraint violations due to cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative offline distributional value learning. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile critics to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide exploration. Experiments on safe velocity, safe navigation, and autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive test safety performance, and controlled data collection cost. The results highlight COX-Q as a promising RL method for safety-critical applications.