Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces

TL;DR

This paper proves the convergence of a deep actor-critic reinforcement learning algorithm for continuous-space mean field games and control problems, extending theoretical guarantees and demonstrating results on linear-quadratic examples.

Contribution

It establishes convergence of the actor-critic algorithm in continuous spaces for MFG and MFC, including discretization techniques and extensions to control games.

Findings

Convergence proven for continuous state and action spaces.

Numerical experiments on linear-quadratic problems confirm theoretical results.

Extension of convergence analysis to mean field control games.

Abstract

We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates: one for the value function and the other for the mean field term. In the MFC case, to rigorously identify the limit, we introduce a discretization of the state and action spaces, following the approach used in the finite-space case in [Angiuli et al., 2023b]. The convergence proofs rely on a generalization of the two-timescale framework introduced in [Borkar, 1997]. We further extend our convergence results to Mean Field Control Games, which involve locally cooperative and globally competitive populations. Finally, we present numerical experiments for linear-quadratic problems in one and two dimensions, for which explicit solutions are available.