Analysis and Improvement of Adversarial Training in DQN Agents With Adversarially-Guided Exploration (AGE)

TL;DR

This paper analyzes the robustness of DQN agents under adversarial training, introduces a novel exploration method called AGE to improve training efficiency, and empirically evaluates its effectiveness against traditional exploration strategies.

Contribution

It provides a formal analysis of adversarial training in DQNs and proposes AGE, a new exploration mechanism that enhances training efficiency and robustness.

Findings

AGE outperforms traditional exploration methods in robustness.

Adversarial training improves DQN robustness to state perturbations.

AGE demonstrates better sample efficiency in experiments.

Abstract

This paper investigates the effectiveness of adversarial training in enhancing the robustness of Deep Q-Network (DQN) policies to state-space perturbations. We first present a formal analysis of adversarial training in DQN agents and its performance with respect to the proportion of adversarial perturbations to nominal observations used for training. Next, we consider the sample-inefficiency of current adversarial training techniques, and propose a novel Adversarially-Guided Exploration (AGE) mechanism based on a modified hybrid of the $\epsilon$-greedy algorithm and Boltzmann exploration. We verify the feasibility of this exploration mechanism through experimental evaluation of its performance in comparison with the traditional decaying $\epsilon$-greedy and parameter-space noise exploration algorithms.