Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning

TL;DR

This paper introduces a mean-field reinforcement learning algorithm called MF-FQI that leverages agent symmetry and mean embedding techniques, providing theoretical guarantees and demonstrating improved performance with more agents.

Contribution

It proposes the MF-FQI algorithm for mean-field MARL, offering a non-asymptotic analysis and revealing a 'blessing of many agents' effect.

Findings

MF-FQI achieves provable convergence.

More agents lead to better performance.

The method handles continuous mean-field states.

Abstract

Multi-agent reinforcement learning (MARL) achieves significant empirical successes. However, MARL suffers from the curse of many agents. In this paper, we exploit the symmetry of agents in MARL. In the most generic form, we study a mean-field MARL problem. Such a mean-field MARL is defined on mean-field states, which are distributions that are supported on continuous space. Based on the mean embedding of the distributions, we propose MF-FQI algorithm that solves the mean-field MARL and establishes a non-asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.