Adversary Agnostic Robust Deep Reinforcement Learning

TL;DR

This paper introduces an adversary agnostic approach to enhance the robustness of deep reinforcement learning policies without relying on adversarial training, using a novel policy distillation method grounded in theoretical guarantees.

Contribution

The paper proposes a new adversary agnostic robust DRL paradigm based on policy distillation, with a novel loss function and theoretical analysis ensuring increased robustness without adversary knowledge.

Findings

Outperforms state-of-the-art methods in Atari games

Theoretically guarantees increased robustness

Effective against various perturbations

Abstract

Deep reinforcement learning (DRL) policies have been shown to be deceived by perturbations (e.g., random noise or intensional adversarial attacks) on state observations that appear at test time but are unknown during training. To increase the robustness of DRL policies, previous approaches assume that the knowledge of adversaries can be added into the training process to achieve the corresponding generalization ability on these perturbed observations. However, such an assumption not only makes the robustness improvement more expensive but may also leave a model less effective to other kinds of attacks in the wild. In contrast, we propose an adversary agnostic robust DRL paradigm that does not require learning from adversaries. To this end, we first theoretically derive that robustness could indeed be achieved independently of the adversaries based on a policy distillation setting. Motivated by this finding, we propose a new policy distillation loss with two terms: 1) a prescription gap maximization loss aiming at simultaneously maximizing the likelihood of the action selected by the teacher policy and the entropy over the remaining actions; 2) a corresponding Jacobian regularization loss that minimizes the magnitude of the gradient with respect to the input state. The theoretical analysis shows that our distillation loss guarantees to increase the prescription gap and the adversarial robustness. Furthermore, experiments on five Atari games firmly verify the superiority of our approach in terms of boosting adversarial robustness compared to other state-of-the-art methods.