Taming the Curses of Multiagency in Robust Markov Games with Large State Space through Linear Function Approximation

TL;DR

This paper introduces data-efficient algorithms for large-scale distributionally robust Markov games using linear function approximation, overcoming the curse of multiagency in complex environments.

Contribution

It develops the first provably data-efficient algorithms for large or infinite state space RMGs with LFA, addressing robustness and sample complexity challenges.

Findings

Algorithms work in both generative and online settings.

First to break the curse of multiagency for large-scale RMGs.

Applicable to uncertainty sets defined by total variation distance.

Abstract

Multi-agent reinforcement learning (MARL) holds great potential but faces robustness challenges due to environmental uncertainty. To address this, distributionally robust Markov games (RMGs) optimize worst-case performance when the environment deviates from the nominal model within a uncertainty set. Beyond robustness, an equally urgent goal for MARL is data efficiency -- sampling from vast state and action spaces that grow exponentially with the number of agents potentially leads to the curse of multiagency. However, current provably data-efficient algorithms for RMGs are limited to tabular settings with finite state and action spaces, which are only computationally manageable for small-scale problems, leaving RMGs with large-scale (or infinite) state spaces largely unexplored. The only existing work beyond tabular settings focuses on linear function approximation (LFA) for a restrictive class of RMGs using vanish minimal value assumption and still suffers from sample complexity with the curse of multiagency. In this work, we focuses on general RMGs with LFA. For uncertainty sets defined by total variation distance, we develop provably data-efficient algorithms that break the curse of multiagency in both the generative model setting and a newly proposed online interactive setting. To our knowledge, our results are the first to break the curse of multiagency of sample complexity for RMGs with large (possibly infinite) state spaces, regardless of the uncertainty set construction.