Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables

TL;DR

This paper introduces GDR-MDP, a hierarchical framework for multi-task reinforcement learning that balances robustness and average performance by modeling task groups with latent variables, leading to improved robustness in diverse environments.

Contribution

The paper proposes GDR-MDP, a novel hierarchical MDP model that encodes task groups via latent variables and develops deep RL algorithms for it, enhancing robustness under task ambiguity.

Findings

GDR-MDP improves distributional robustness through hierarchical regularization.

Algorithms outperform classic robust training in diverse benchmarks.

Demonstrated effectiveness on Box2D, MuJoCo, and Google football platforms.

Abstract

One key challenge for multi-task Reinforcement learning (RL) in practice is the absence of task indicators. Robust RL has been applied to deal with task ambiguity, but may result in over-conservative policies. To balance the worst-case (robustness) and average performance, we propose Group Distributionally Robust Markov Decision Process (GDR-MDP), a flexible hierarchical MDP formulation that encodes task groups via a latent mixture model. GDR-MDP identifies the optimal policy that maximizes the expected return under the worst-possible qualified belief over task groups within an ambiguity set. We rigorously show that GDR-MDP's hierarchical structure improves distributional robustness by adding regularization to the worst possible outcomes. We then develop deep RL algorithms for GDR-MDP for both value-based and policy-based RL methods. Extensive experiments on Box2D control tasks, MuJoCo benchmarks, and Google football platforms show that our algorithms outperform classic robust training algorithms across diverse environments in terms of robustness under belief uncertainties. Demos are available on our project page (\url{https://sites.google.com/view/gdr-rl/home}).