The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

TL;DR

This paper introduces a provable algorithm for multi-agent reinforcement learning in large state spaces, leveraging a new complexity measure and an exploiter component to find Nash equilibria efficiently.

Contribution

It extends RL theory to multi-agent settings with function approximation, introducing the multi-agent Bellman-Eluder dimension and a novel exploiter-based algorithm.

Findings

Algorithm finds Nash equilibrium with polynomial samples

Applicable to various MG models including linear and kernel approximations

Provides theoretical guarantees in large state spaces

Abstract

Modern reinforcement learning (RL) commonly engages practical problems with large state spaces, where function approximation must be deployed to approximate either the value function or the policy. While recent progresses in RL theory address a rich set of RL problems with general function approximation, such successes are mostly restricted to the single-agent setting. It remains elusive how to extend these results to multi-agent RL, especially due to the new challenges arising from its game-theoretical nature. This paper considers two-player zero-sum Markov Games (MGs). We propose a new algorithm that can provably find the Nash equilibrium policy using a polynomial number of samples, for any MG with low multi-agent Bellman-Eluder dimension -- a new complexity measure adapted from its single-agent version (Jin et al., 2021). A key component of our new algorithm is the exploiter, which facilitates the learning of the main player by deliberately exploiting her weakness. Our theoretical framework is generic, which applies to a wide range of models including but not limited to tabular MGs, MGs with linear or kernel function approximation, and MGs with rich observations.