A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

TL;DR

This paper introduces a unified game-theoretic framework for multiagent reinforcement learning that improves policy generalization and scalability, demonstrated in gridworld and poker environments.

Contribution

It proposes a new algorithm combining deep RL, empirical game theory, and meta-strategies, unifying and extending previous MARL methods.

Findings

Policies overfit to training partners, reducing generalization.

The new algorithm outperforms previous methods in complex environments.

Scalable implementation reduces memory requirements.

Abstract

To achieve general intelligence, agents must learn how to interact with others in a shared environment: this is the challenge of multiagent reinforcement learning (MARL). The simplest form is independent reinforcement learning (InRL), where each agent treats its experience as part of its (non-stationary) environment. In this paper, we first observe that policies learned using InRL can overfit to the other agents' policies during training, failing to sufficiently generalize during execution. We introduce a new metric, joint-policy correlation, to quantify this effect. We describe an algorithm for general MARL, based on approximate best responses to mixtures of policies generated using deep reinforcement learning, and empirical game-theoretic analysis to compute meta-strategies for policy selection. The algorithm generalizes previous ones such as InRL, iterated best response, double oracle, and fictitious play. Then, we present a scalable implementation which reduces the memory requirement using decoupled meta-solvers. Finally, we demonstrate the generality of the resulting policies in two partially observable settings: gridworld coordination games and poker.