Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

TL;DR

This paper develops and analyzes q-learning algorithms for mean-field control problems with common noise, including convergence proofs and practical implementations in linear quadratic and other settings.

Contribution

It introduces new Actor-Critic q-learning algorithms for mean-field control with common noise, incorporating error quantification and convergence analysis.

Findings

Algorithms demonstrate satisfactory performance in examples.

Error bounds are established for observable data approximation.

Convergence of inner iterations is proven in the LQ framework.

Abstract

This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.