Investigating Vulnerabilities of Deep Neural Policies

TL;DR

This paper investigates how adversarial training affects deep neural policies in reinforcement learning, revealing increased sensitivity to low-frequency perturbations and differences in feature sensitivities, contributing to understanding robustness.

Contribution

It introduces a novel analysis of adversarial training effects on neural policies through Fourier spectrum and feature sensitivity comparisons.

Findings

Adversarially trained policies are more sensitive to low-frequency perturbations.

Fourier analysis shows a focus on lower frequencies in adversarially trained policies.

Feature sensitivity differences highlight robustness variations between training methods.

Abstract

Reinforcement learning policies based on deep neural networks are vulnerable to imperceptible adversarial perturbations to their inputs, in much the same way as neural network image classifiers. Recent work has proposed several methods to improve the robustness of deep reinforcement learning agents to adversarial perturbations based on training in the presence of these imperceptible perturbations (i.e. adversarial training). In this paper, we study the effects of adversarial training on the neural policy learned by the agent. In particular, we follow two distinct parallel approaches to investigate the outcomes of adversarial training on deep neural policies based on worst-case distributional shift and feature sensitivity. For the first approach, we compare the Fourier spectrum of minimal perturbations computed for both adversarially trained and vanilla trained neural policies. Via experiments in the OpenAI Atari environments we show that minimal perturbations computed for adversarially trained policies are more focused on lower frequencies in the Fourier domain, indicating a higher sensitivity of these policies to low frequency perturbations. For the second approach, we propose a novel method to measure the feature sensitivities of deep neural policies and we compare these feature sensitivity differences in state-of-the-art adversarially trained deep neural policies and vanilla trained deep neural policies. We believe our results can be an initial step towards understanding the relationship between adversarial training and different notions of robustness for neural policies.