Vulnerability Analysis of Safe Reinforcement Learning via Inverse Constrained Reinforcement Learning

TL;DR

This paper presents a black-box adversarial attack framework that exposes vulnerabilities in safe reinforcement learning policies by using expert demonstrations and surrogate models, without needing internal policy details.

Contribution

It introduces a novel attack method that operates with limited access, combining constraint modeling and surrogate policies to challenge Safe RL systems.

Findings

Effective attacks on Safe RL benchmarks under limited access

Theoretical bounds on perturbations and attack feasibility

Demonstrated vulnerability of Safe RL in realistic scenarios

Abstract

Safe reinforcement learning (Safe RL) aims to ensure policy performance while satisfying safety constraints. However, most existing Safe RL methods assume benign environments, making them vulnerable to adversarial perturbations commonly encountered in real-world settings. In addition, existing gradient-based adversarial attacks typically require access to the policy's gradient information, which is often impractical in real-world scenarios. To address these challenges, we propose an adversarial attack framework to reveal vulnerabilities of Safe RL policies. Using expert demonstrations and black-box environment interaction, our framework learns a constraint model and a surrogate (learner) policy, enabling gradient-based attack optimization without requiring the victim policy's internal gradients or the ground-truth safety constraints. We further provide theoretical analysis establishing feasibility and deriving perturbation bounds. Experiments on multiple Safe RL benchmarks demonstrate the effectiveness of our approach under limited privileged access.