Robust Deep Reinforcement Learning against Adversarial Behavior Manipulation

TL;DR

This paper introduces a novel environment-agnostic attack method on reinforcement learning using imitation learning, and proposes a new defense strategy based on time-discounted regularization to improve robustness against behavior-targeted manipulations.

Contribution

It presents the first defense specifically designed for behavior-targeted attacks and offers a new attack method that requires limited access to the victim's policy.

Findings

The proposed attack method works under limited policy access.

Time-discounted regularization improves robustness against attacks.

Theoretical analysis links policy sensitivity to defense effectiveness.

Abstract

This study investigates behavior-targeted attacks on reinforcement learning and their countermeasures. Behavior-targeted attacks aim to manipulate the victim's behavior as desired by the adversary through adversarial interventions in state observations. Existing behavior-targeted attacks have some limitations, such as requiring white-box access to the victim's policy. To address this, we propose a novel attack method using imitation learning from adversarial demonstrations, which works under limited access to the victim's policy and is environment-agnostic. In addition, our theoretical analysis proves that the policy's sensitivity to state changes impacts defense performance, particularly in the early stages of the trajectory. Based on this insight, we propose time-discounted regularization, which enhances robustness against attacks while maintaining task performance. To the best of our knowledge, this is the first defense strategy specifically designed for behavior-targeted attacks.