Robust Deep Reinforcement Learning with Adversarial Attacks

TL;DR

This paper introduces adversarial attacks on Deep Reinforcement Learning algorithms and uses them to enhance robustness against parameter uncertainties through adversarial training, improving performance on standard benchmarks.

Contribution

It develops effective adversarial attacks for RL and demonstrates their use in adversarial training to significantly improve robustness of DRL algorithms.

Findings

Adversarial attacks successfully degrade DRL performance.

Gradient-based attacks further increase degradation.

Adversarial training enhances robustness on multiple RL benchmarks.

Abstract

This paper proposes adversarial attacks for Reinforcement Learning (RL) and then improves the robustness of Deep Reinforcement Learning algorithms (DRL) to parameter uncertainties with the help of these attacks. We show that even a naively engineered attack successfully degrades the performance of DRL algorithm. We further improve the attack using gradient information of an engineered loss function which leads to further degradation in performance. These attacks are then leveraged during training to improve the robustness of RL within robust control framework. We show that this adversarial training of DRL algorithms like Deep Double Q learning and Deep Deterministic Policy Gradients leads to significant increase in robustness to parameter variations for RL benchmarks such as Cart-pole, Mountain Car, Hopper and Half Cheetah environment.