Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise

TL;DR

This paper provides a comprehensive analysis of policy gradient algorithms for continuous-time linear quadratic mean-field control problems with common noise, demonstrating convergence, sample complexity, and numerical validation.

Contribution

It offers the first global linear convergence and sample complexity analysis of model-free policy gradient methods for mean-field control with common noise.

Findings

Proves global linear convergence of policy gradient algorithms.

Establishes sample complexity bounds in a model-free setting.

Numerical experiments support theoretical convergence results.

Abstract

We consider reinforcement learning (RL) methods for finding optimal policies in linear quadratic (LQ) mean field control (MFC) problems over an infinite horizon in continuous time, with common noise and entropy regularization. We study policy gradient (PG) learning and first demonstrate convergence in a model-based setting by establishing a suitable gradient domination condition.Next, our main contribution is a comprehensive error analysis, where we prove the global linear convergence and sample complexity of the PG algorithm with two-point gradient estimates in a model-free setting with unknown parameters. In this setting, the parameterized optimal policies are learned from samples of the states and population distribution.Finally, we provide numerical evidence supporting the convergence of our implemented algorithms.