Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective

TL;DR

This paper develops a mean-field game framework for robust cooperative multi-agent reinforcement learning under uncertainty, providing theoretical guarantees and a practical algorithm with convergence analysis.

Contribution

It introduces a mean-field type game approach to address robustness in multi-agent RL with distributed information, and proposes a convergent gradient-based algorithm.

Findings

The proposed method achieves near-optimal Nash equilibrium in simulations.

The algorithm converges at a non-asymptotic rate.

Numerical results outperform baseline algorithms.

Abstract

In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of \emph{stochastic} and \emph{non-stochastic} uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem in a worst-case (minimax) framework, which is is intractable in general. Thus, we focus on the Linear Quadratic setting to derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.