Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

TL;DR

This paper introduces RSPG, a novel history-aware hybrid structural method for partially observable mean field games, achieving faster convergence and enabling complex macroeconomic modeling with heterogeneous agents and common noise.

Contribution

The paper presents RSPG, the first history-aware hybrid structural method for partially observable MFGs, and MFAX, a framework that leverages known dynamics for improved performance.

Findings

RSPG achieves state-of-the-art performance.

RSPG converges an order of magnitude faster.

Successfully models macroeconomics MFG with heterogeneity and common noise.

Abstract

Mean Field Games (MFGs) provide a principled framework for modeling interactions in large population models: at scale, population dynamics become deterministic, with uncertainty entering only through aggregate shocks, or common noise. However, algorithmic progress has been limited since model-free methods are too high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) use Monte Carlo rollouts for the common noise in combination with exact estimation of the expected return, conditioned on those samples. However, HSMs have not been scaled to Partially Observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for settings involving public information. We also introduce MFAX, our JAX-based framework for MFGs. By leveraging known transition dynamics, RSPG achieves state-of-the-art performance as well as an order-of-magnitude faster convergence and solves, for the first time, a macroeconomics MFG with heterogeneous agents, common noise and history-aware policies. MFAX is publicly available at: https://github.com/CWibault/mfax.