Learning optimal policies in potential Mean Field Games: Smoothed Policy Iteration algorithms

TL;DR

This paper introduces two Smoothed Policy Iteration algorithms for learning Nash equilibria in second order potential Mean Field Games, with proven convergence properties and numerical validation.

Contribution

The paper proposes novel Smoothed Policy Iteration algorithms for potential MFGs, establishing convergence under certain conditions and linking to Fictitious Play.

Findings

Global convergence under monotonicity condition

Local convergence for multiple solutions

Numerical simulations validate theoretical results

Abstract

We introduce two Smoothed Policy Iteration algorithms (\textbf{SPI}s) as rules for learning policies and methods for computing Nash equilibria in second order potential Mean Field Games (MFGs). Global convergence is proved if the coupling term in the MFG system satisfy the Lasry Lions monotonicity condition. Local convergence to a stable solution is proved for system which may have multiple solutions. The convergence analysis shows close connections between \textbf{SPI}s and the Fictitious Play algorithm, which has been widely studied in the MFG literature. Numerical simulation results based on finite difference schemes are presented to supplement the theoretical analysis.