Diffusion Guided Adversarial State Perturbations in Reinforcement Learning

TL;DR

This paper introduces SHIFT, a diffusion-based attack method that creates semantically meaningful adversarial state perturbations in reinforcement learning, exposing vulnerabilities of current defenses and emphasizing the need for more robust policies.

Contribution

The paper presents SHIFT, a novel diffusion-guided attack that surpasses traditional norm-based attacks by generating realistic, semantics-altering adversarial states in RL environments.

Findings

SHIFT effectively breaks existing defenses against RL attacks.

The attack produces more perceptually stealthy adversarial states.

Current defenses are vulnerable to semantics-aware perturbations.

Abstract

Reinforcement learning (RL) systems, while achieving remarkable success across various domains, are vulnerable to adversarial attacks. This is especially a concern in vision-based environments where minor manipulations of high-dimensional image inputs can easily mislead the agent's behavior. To this end, various defenses have been proposed recently, with state-of-the-art approaches achieving robust performance even under large state perturbations. However, after closer investigation, we found that the effectiveness of the current defenses is due to a fundamental weakness of the existing $l_p$ norm-constrained attacks, which can barely alter the semantics of image input even under a relatively large perturbation budget. In this work, we propose SHIFT, a novel policy-agnostic diffusion-based state perturbation attack to go beyond this limitation. Our attack is able to generate perturbed states that are semantically different from the true states while remaining realistic and history-aligned to avoid detection. Evaluations show that our attack effectively breaks existing defenses, including the most sophisticated ones, significantly outperforming existing attacks while being more perceptually stealthy. The results highlight the vulnerability of RL agents to semantics-aware adversarial perturbations, indicating the importance of developing more robust policies.