RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors

TL;DR

This paper introduces RAT, a novel adversarial attack method on deep reinforcement learning agents that precisely manipulates targeted behaviors, surpassing existing methods in effectiveness and robustness across robotic and MuJoCo tasks.

Contribution

RAT is the first method to enable universal, targeted behavior attacks on DRL agents by aligning an intention policy with human preferences and dynamically adjusting state occupancy measures.

Findings

RAT outperforms existing attack algorithms in robotic simulations.

RAT effectively guides agents to adopt human-aligned behaviors in MuJoCo tasks.

RAT enhances the robustness of DRL agents against targeted behavior manipulations.

Abstract

Evaluating deep reinforcement learning (DRL) agents against targeted behavior attacks is critical for assessing their robustness. These attacks aim to manipulate the victim into specific behaviors that align with the attacker's objectives, often bypassing traditional reward-based defenses. Prior methods have primarily focused on reducing cumulative rewards; however, rewards are typically too generic to capture complex safety requirements effectively. As a result, focusing solely on reward reduction can lead to suboptimal attack strategies, particularly in safety-critical scenarios where more precise behavior manipulation is needed. To address these challenges, we propose RAT, a method designed for universal, targeted behavior attacks. RAT trains an intention policy that is explicitly aligned with human preferences, serving as a precise behavioral target for the adversary. Concurrently, an adversary manipulates the victim's policy to follow this target behavior. To enhance the effectiveness of these attacks, RAT dynamically adjusts the state occupancy measure within the replay buffer, allowing for more controlled and effective behavior manipulation. Our empirical results on robotic simulation tasks demonstrate that RAT outperforms existing adversarial attack algorithms in inducing specific behaviors. Additionally, RAT shows promise in improving agent robustness, leading to more resilient policies. We further validate RAT by guiding Decision Transformer agents to adopt behaviors aligned with human preferences in various MuJoCo tasks, demonstrating its effectiveness across diverse tasks.