Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning

TL;DR

This paper introduces a new framework for understanding and generating optimal adversarial attacks on deep reinforcement learning models by exploring environmental dynamics and reformulating attack strategies in function space.

Contribution

It presents a two-stage framework that trains deceptive policies and generates stronger adversarial attacks, outperforming existing methods in efficiency and effectiveness.

Findings

Achieves state-of-the-art results in Atari and MuJoCo environments.

Theoretically demonstrates the superiority of the proposed adversary.

Outperforms existing optimization-based attack methods.

Abstract

Deep reinforcement learning models are vulnerable to adversarial attacks that can decrease a victim's cumulative expected reward by manipulating the victim's observations. Despite the efficiency of previous optimization-based methods for generating adversarial noise in supervised learning, such methods might not be able to achieve the lowest cumulative reward since they do not explore the environmental dynamics in general. In this paper, we provide a framework to better understand the existing methods by reformulating the problem of adversarial attacks on reinforcement learning in the function space. Our reformulation generates an optimal adversary in the function space of the targeted attacks, repelling them via a generic two-stage framework. In the first stage, we train a deceptive policy by hacking the environment, and discover a set of trajectories routing to the lowest reward or the worst-case performance. Next, the adversary misleads the victim to imitate the deceptive policy by perturbing the observations. Compared to existing approaches, we theoretically show that our adversary is stronger under an appropriate noise level. Extensive experiments demonstrate our method's superiority in terms of efficiency and effectiveness, achieving the state-of-the-art performance in both Atari and MuJoCo environments.