Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration

TL;DR

This paper introduces MICE, a novel method for constrained reinforcement learning that mitigates underestimation bias by leveraging a memory module of unsafe states, leading to safer exploration and fewer constraint violations.

Contribution

The paper proposes the Memory-driven Intrinsic Cost Estimation (MICE) method, which uses a memory module and bias correction to improve safety in constrained reinforcement learning.

Findings

MICE reduces constraint violations significantly.

MICE maintains comparable policy performance to baselines.

Theoretical guarantees for convergence and constraint violation bounds.

Abstract

Constrained Reinforcement Learning (CRL) aims to maximize cumulative rewards while satisfying constraints. However, existing CRL algorithms often encounter significant constraint violations during training, limiting their applicability in safety-critical scenarios. In this paper, we identify the underestimation of the cost value function as a key factor contributing to these violations. To address this issue, we propose the Memory-driven Intrinsic Cost Estimation (MICE) method, which introduces intrinsic costs to mitigate underestimation and control bias to promote safer exploration. Inspired by flashbulb memory, where humans vividly recall dangerous experiences to avoid risks, MICE constructs a memory module that stores previously explored unsafe states to identify high-cost regions. The intrinsic cost is formulated as the pseudo-count of the current state visiting these risk regions. Furthermore, we propose an extrinsic-intrinsic cost value function that incorporates intrinsic costs and adopts a bias correction strategy. Using this function, we formulate an optimization objective within the trust region, along with corresponding optimization methods. Theoretically, we provide convergence guarantees for the proposed cost value function and establish the worst-case constraint violation for the MICE update. Extensive experiments demonstrate that MICE significantly reduces constraint violations while preserving policy performance comparable to baselines.