Reinforcement Learning for Finite Space Mean-Field Type Games

TL;DR

This paper introduces scalable reinforcement learning algorithms for finite space mean-field type games, providing theoretical guarantees and demonstrating effectiveness in high-dimensional environments.

Contribution

It develops the first scalable RL methods for finite space MFTGs with convergence analysis and practical algorithms including Nash Q-learning and deep RL.

Findings

Algorithms scale to mean-field distributions of dimension up to 200.

Proposed methods are efficient and scalable in numerical experiments.

Theoretical analysis confirms convergence and stability.

Abstract

Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that the MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on the quantization of mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose a deep reinforcement learning algorithm, which can scale to larger spaces. Numerical experiments in 4 environments with mean-field distributions of dimension up to $200$ show the scalability and efficiency of the proposed method.