Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time

TL;DR

This paper proves that policy gradient methods in continuous-time linear-quadratic mean-field control and game settings converge linearly to optimal solutions, providing theoretical guarantees and conditions for Nash equilibrium existence.

Contribution

It establishes the first convergence analysis of policy gradient algorithms in continuous-time linear-quadratic mean-field models, including conditions for Nash equilibrium.

Findings

Policy gradient converges linearly to the optimal in MFC and MFG.

Provided sufficient conditions for Nash equilibrium existence and uniqueness.

Validated results through synthetic simulations.

Abstract

Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment. As the number of agents grow to be very large, the system can be approximated by a mean-field problem. Therefore, it has motivated new research directions for mean-field control (MFC) and mean-field game (MFG). In this paper, we study the policy gradient method for the linear-quadratic mean-field control and game, where we assume each agent has identical linear state transitions and quadratic cost functions. While most of the recent works on policy gradient for MFC and MFG are based on discrete-time models, we focus on the continuous-time models where some analyzing techniques can be interesting to the readers. For both MFC and MFG, we provide policy gradient update and show that it converges to the optimal solution at a linear rate, which is verified by a synthetic simulation. For MFG, we also provide sufficient conditions for the existence and uniqueness of the Nash equilibrium.