Mean Field LQG Social Optimization: A Reinforcement Learning Approach

TL;DR

This paper introduces a model-free reinforcement learning method to solve mean field LQG social control problems with multiplicative noise, eliminating the need for prior knowledge of individual dynamics.

Contribution

It develops a unified, scalable RL-based approach to solve stochastic Riccati equations and estimate the mean field state using data from a single agent.

Findings

Converges to solutions of stochastic AREs without prior system knowledge

Uses data from a single agent for multiple steps, improving efficiency

Demonstrates effectiveness through a numerical example

Abstract

This paper presents a novel model-free method to solve linear quadratic Gaussian mean field social control problems in the presence of multiplicative noise. The objective is to achieve a social optimum by solving two algebraic Riccati equations (AREs) and determining a mean field (MF) state, both without requiring prior knowledge of individual system dynamics for all agents. In the proposed approach, we first employ integral reinforcement learning techniques to develop two model-free iterative equations that converge to solutions for the stochastic ARE and the induced indefinite ARE respectively. Then, the MF state is approximated, either through the Monte Carlo method with the obtained gain matrices or through the system identification with the measured data. Notably, a unified state and input samples collected from a single agent are used in both iterations and identification procedure, making the method more computationally efficient and scalable. Finally, a numerical example is given to demonstrate the effectiveness of the proposed algorithm.