A Game-Theoretic Perspective of Generalization in Reinforcement Learning

TL;DR

This paper introduces a game-theoretic framework called GiRL for analyzing and improving generalization in reinforcement learning by training agents against adversaries manipulating task distributions, unifying various schemes.

Contribution

It proposes a unified game-theoretic framework for RL generalization and adapts PSRO with novel modifications for effective training and testing strategies.

Findings

Outperforms existing baselines like MAML in MuJoCo environments.

Unifies multiple RL generalization schemes under a single framework.

Demonstrates the effectiveness of the proposed methods through extensive experiments.

Abstract

Generalization in reinforcement learning (RL) is of importance for real deployment of RL algorithms. Various schemes are proposed to address the generalization issues, including transfer learning, multi-task learning and meta learning, as well as the robust and adversarial reinforcement learning. However, there is not a unified formulation of the various schemes, as well as the comprehensive comparisons of methods across different schemes. In this work, we propose a game-theoretic framework for the generalization in reinforcement learning, named GiRL, where an RL agent is trained against an adversary over a set of tasks, where the adversary can manipulate the distributions over tasks within a given threshold. With different configurations, GiRL can reduce the various schemes mentioned above. To solve GiRL, we adapt the widely-used method in game theory, policy space response oracle (PSRO) with the following three important modifications: i) we use model-agnostic meta learning (MAML) as the best-response oracle, ii) we propose a modified projected replicated dynamics, i.e., R-PRD, which ensures the computed meta-strategy of the adversary fall in the threshold, and iii) we also propose a protocol for the few-shot learning of the multiple strategies during testing. Extensive experiments on MuJoCo environments demonstrate that our proposed methods can outperform existing baselines, e.g., MAML.