Wasserstein Robust Reinforcement Learning

TL;DR

This paper introduces $ ext{W} ext{R}^{2} ext{L}$, a Wasserstein-based robust reinforcement learning algorithm that improves performance and robustness in complex control tasks through a novel min-max formulation and scalable solver.

Contribution

It formulates robust reinforcement learning as a Wasserstein-constrained min-max game and proposes a new zero-order optimization method for efficient solving.

Findings

Significant performance gains over standard algorithms in MuJuCo environments.

Robustness to environment variations demonstrated in high-dimensional tasks.

Scalable solver applicable to complex reinforcement learning problems.

Abstract

Reinforcement learning algorithms, though successful, tend to over-fit to training environments hampering their application to the real-world. This paper proposes $\text{W}\text{R}^{2}\text{L}$ -- a robust reinforcement learning algorithm with significant robust performance on low and high-dimensional control tasks. Our method formalises robust reinforcement learning as a novel min-max game with a Wasserstein constraint for a correct and convergent solver. Apart from the formulation, we also propose an efficient and scalable solver following a novel zero-order optimisation method that we believe can be useful to numerical optimisation in general. We empirically demonstrate significant gains compared to standard and robust state-of-the-art algorithms on high-dimensional MuJuCo environments.