Robust Constrained Reinforcement Learning

TL;DR

This paper introduces a robust constrained reinforcement learning framework that ensures constraint satisfaction and maximizes worst-case reward under model uncertainty, addressing issues like modeling errors and adversarial attacks.

Contribution

It proposes a novel robust primal-dual approach with theoretical guarantees and develops an online, model-free algorithm for $$-contamination uncertainty sets.

Findings

Guarantees on convergence, complexity, and robust feasibility of the proposed method.

An online, model-free algorithm with characterized sample complexity.

Effective handling of model uncertainty in constrained RL scenarios.

Abstract

Constrained reinforcement learning is to maximize the expected reward subject to constraints on utilities/costs. However, the training environment may not be the same as the test one, due to, e.g., modeling error, adversarial attack, non-stationarity, resulting in severe performance degradation and more importantly constraint violation. We propose a framework of robust constrained reinforcement learning under model uncertainty, where the MDP is not fixed but lies in some uncertainty set, the goal is to guarantee that constraints on utilities/costs are satisfied for all MDPs in the uncertainty set, and to maximize the worst-case reward performance over the uncertainty set. We design a robust primal-dual approach, and further theoretically develop guarantee on its convergence, complexity and robust feasibility. We then investigate a concrete example of $\delta$-contamination uncertainty set, design an online and model-free algorithm and theoretically characterize its sample complexity.