Deep Policy Iteration for High-Dimensional Mean-Field Games with Regenerative Reformulation

TL;DR

This paper introduces a scalable deep policy iteration approach for high-dimensional mean-field games, reformulating the problem as a regenerative process to improve efficiency and handle dimensions up to 10,000.

Contribution

The paper proposes a novel regenerative reformulation and deep policy iteration method that significantly enhances scalability for high-dimensional mean-field games.

Findings

Effective handling of dimensions up to 10,000

Avoids solving coupled HJB and Fokker-Planck equations

Uses particle system and adversarial training for efficiency

Abstract

This paper develops a deep policy iteration method for high-dimensional finite-horizon mean-field games (MFG). We reformulate the game as a regenerative problem with deterministic cycles, which allows policy evaluation (PE), policy improvement (PI), and population measure estimation to be carried out cycle by cycle. Within this formulation, we approximate the population measure by a particle system and update it using a one-step random mapping induced by the Euler-Maruyama discretization of the state dynamics. This update transports a mini-batch of particles from one cycle to the next, avoiding sequential trajectory simulation over the entire time horizon at each iteration. The PE and PI subproblems are formulated through the relation between consecutive cycles, with adversarial training used for evaluation and averaged optimization used for improvement. The resulting method is efficient and scalable in high dimensions, as it avoids the direct solution of the coupled Hamilton-Jacobi-Bellman and Fokker-Planck system, the full simulation of trajectories to estimate the population measure, the explicit computation of conditional expectations in policy evaluation, and pointwise optimization in policy improvement. Numerical experiments demonstrate that the proposed method effectively handles dimensions up to 10,000.