Robust Mean Field Social Control: A Unified Reinforcement Learning Framework

TL;DR

This paper introduces a robust, data-driven reinforcement learning framework for mean field social control problems with multiplicative noise, addressing challenges in solving indefinite Riccati equations and ensuring convergence and robustness.

Contribution

It develops a unified dual-loop iterative algorithm for indefinite Riccati equations, proves its convergence and robustness, and incorporates integral reinforcement learning for a data-driven approach.

Findings

Algorithm converges to a neighborhood of the optimal solution.

The method is robust to estimation and modeling errors.

Numerical example demonstrates effectiveness of the approach.

Abstract

This paper studies linear quadratic Gaussian robust mean field social control problems in the presence of multiplicative noise. We aim to compute asymptotic decentralized strategies without requiring full prior knowledge of agents' dynamics. The primary challenges lie in solving an indefinite stochastic algebraic Riccati equation for feedback gains, and an indefinite algebraic Riccati equation for feedforward gains. To overcome these challenges, we first propose a unified dual-loop iterative framework that handles both indefinite Riccati-type equations, and provide rigorous convergence proofs for both the outer-loop and inner-loop iterations. Secondly, considering the potential biases arising in the iterative processes due to estimation and modeling errors, we verify the robustness of the proposed algorithm using the small-disturbance input-to-state stability technique. Convergence to a neighborhood of the optimal solution is thus ensured, even in the existence of disturbances. Finally, to relax the limitation of requiring precise knowledge of agents' dynamics, we employ the integral reinforcement learning technique to develop a data-driven method within the dual-loop iterative framework. A numerical example is provided to demonstrate the effectiveness of the proposed algorithm.