Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

TL;DR

This paper analyzes the convergence of policy gradient methods in mean-field reinforcement learning, specifically in linear-quadratic settings, with theoretical proofs and empirical validation for large-scale multi-agent systems.

Contribution

It provides the first rigorous convergence analysis of policy gradient algorithms in mean-field linear-quadratic reinforcement learning, including rate bounds and empirical demonstrations.

Findings

Proved convergence of policy gradient methods in mean-field LQ setting.

Established bounds on convergence rates.

Provided empirical evidence supporting theoretical results.

Abstract

We investigate reinforcement learning in the setting of Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Applications include, for example, the control of a large number of robots communicating through a central unit dispatching the optimal policy computed by maximizing an aggregate reward. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states and actions of the other agents. We first provide a full analysis this discrete-time mean field control problem. We then rigorously prove the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting and establish bounds on the rates of convergence. We also provide graphical evidence of the convergence based on implementations of our algorithms.