Adversarial Style Transfer for Robust Policy Optimization in Deep Reinforcement Learning

TL;DR

This paper introduces ARPO, an adversarial style transfer method that enhances the robustness and generalization of deep reinforcement learning policies by actively perturbing observations during training.

Contribution

It proposes a novel max-min game theoretic framework with a generator and policy network to improve policy robustness against confounding features.

Findings

ARPO outperforms baseline algorithms on Procgen and Distracting Control Suite.

The method improves generalization to unseen environments.

ARPO enhances sample efficiency in reinforcement learning.

Abstract

This paper proposes an algorithm that aims to improve generalization for reinforcement learning agents by removing overfitting to confounding features. Our approach consists of a max-min game theoretic objective. A generator transfers the style of observation during reinforcement learning. An additional goal of the generator is to perturb the observation, which maximizes the agent's probability of taking a different action. In contrast, a policy network updates its parameters to minimize the effect of such perturbations, thus staying robust while maximizing the expected future reward. Based on this setup, we propose a practical deep reinforcement learning algorithm, Adversarial Robust Policy Optimization (ARPO), to find a robust policy that generalizes to unseen environments. We evaluate our approach on Procgen and Distracting Control Suite for generalization and sample efficiency. Empirically, ARPO shows improved performance compared to a few baseline algorithms, including data augmentation.