Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

TL;DR

This paper develops a theoretical framework for continuous-time q-learning in mean-field control with common noise, introducing the Iq-function and establishing optimal policy characterizations.

Contribution

It introduces the Iq-function in mean-field control, derives the exploratory HJB equation, and proves existence and uniqueness of optimal policies under certain conditions.

Findings

Convergence of value functions under discretized actions to relaxed control formulation.

Derivation of the exploratory HJB equation incorporating common noise.

Explicit Gaussian policy characterization in the linear-quadratic setting.

Abstract

This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.