Safe Reinforcement Learning with Dual Robustness

TL;DR

This paper introduces a unified framework and algorithm for safe and robust reinforcement learning, effectively handling adversarial disturbances while ensuring safety and high performance.

Contribution

It proposes a dual policy iteration scheme within a constrained zero-sum Markov game framework, unifying safe RL and robust RL with proven convergence and a practical deep RL algorithm.

Findings

DRAC achieves high performance under various adversarial scenarios.

It outperforms baseline methods significantly in safety-critical benchmarks.

The convergence of the proposed iteration scheme is theoretically established.

Abstract

Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or compromise safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. Optimality is only valid inside a feasible region, while identification of maximal feasible region must rely on learning the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including problem formulation, iteration scheme, convergence analysis and practical algorithm design. This unification is built upon constrained two-player zero-sum Markov games. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. The convergence of this iteration scheme is proved. Furthermore, we design a deep RL algorithm for practical implementation, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines significantly.