Sample-Efficient Distributionally Robust Multi-Agent Reinforcement Learning via Online Interaction

TL;DR

This paper introduces an online learning algorithm for distributionally robust multi-agent reinforcement learning, enabling agents to learn robust policies directly from environment interactions without prior data, with theoretical guarantees.

Contribution

It pioneers online learning in distributionally robust Markov games and proposes the MORNAVI algorithm with provable guarantees for robustness and efficiency.

Findings

Achieves low regret in robust policy learning.

Effectively handles uncertainties measured by TV and KL divergences.

Provides the first theoretical guarantees for online DRMG algorithms.

Abstract

Well-trained multi-agent systems can fail when deployed in real-world environments due to model mismatches between the training and deployment environments, caused by environment uncertainties including noise or adversarial attacks. Distributionally Robust Markov Games (DRMGs) enhance system resilience by optimizing for worst-case performance over a defined set of environmental uncertainties. However, current methods are limited by their dependence on simulators or large offline datasets, which are often unavailable. This paper pioneers the study of online learning in DRMGs, where agents learn directly from environmental interactions without prior data. We introduce the Multiplayer Optimistic Robust Nash Value Iteration (MORNAVI) algorithm and provide the first provable guarantees for this setting. Our theoretical analysis demonstrates that the algorithm achieves low regret and efficiently finds the optimal robust policy for uncertainty sets measured by Total Variation divergence and Kullback-Leibler divergence. These results establish a new, practical path toward developing truly robust multi-agent systems.