Minimax-Optimal Multi-Agent Robust Reinforcement Learning

TL;DR

This paper introduces a minimax-optimal algorithm for multi-agent robust reinforcement learning in finite-horizon Markov games, achieving near-optimal sample complexity for approximating equilibrium solutions under uncertainty.

Contribution

It extends the Q-FTRL algorithm to finite-horizon multi-agent settings with uncertainty, providing tight sample complexity bounds and achieving equilibrium with provable optimality.

Findings

Achieves minimax-optimal sample complexity for robust equilibrium

Extends Q-FTRL algorithm to multi-agent finite-horizon RMGs

Proves optimality via information-theoretic lower bounds

Abstract

Multi-agent robust reinforcement learning, also known as multi-player robust Markov games (RMGs), is a crucial framework for modeling competitive interactions under environmental uncertainties, with wide applications in multi-agent systems. However, existing results on sample complexity in RMGs suffer from at least one of three obstacles: restrictive range of uncertainty level or accuracy, the curse of multiple agents, and the barrier of long horizons, all of which cause existing results to significantly exceed the information-theoretic lower bound. To close this gap, we extend the Q-FTRL algorithm \citep{li2022minimax} to the RMGs in finite-horizon setting, assuming access to a generative model. We prove that the proposed algorithm achieves an $\varepsilon$-robust coarse correlated equilibrium (CCE) with a sample complexity (up to log factors) of $\widetilde{O}\left(H^3S\sum_{i=1}^mA_i\min\left\{H,1/R\right\}/\varepsilon^2\right)$, where $S$ denotes the number of states, $A_i$ is the number of actions of the $i$-th agent, $H$ is the finite horizon length, and $R$ is uncertainty level. We also show that this sample compelxity is minimax optimal by combining an information-theoretic lower bound. Additionally, in the special case of two-player zero-sum RMGs, the algorithm achieves an $\varepsilon$-robust Nash equilibrium (NE) with the same sample complexity.