Initialization-driven neural generation and training for high-dimensional optimal control and first-order mean field games

TL;DR

This paper presents a neural network-based approach for approximating value functions in high-dimensional optimal control and computing equilibria in first-order mean field games, leveraging initialization strategies and coupled neural networks.

Contribution

It introduces a novel initialization-driven neural method for high-dimensional optimal control and MFGs, integrating neural networks with PMP, HJB equations, and fictitious play.

Findings

Effective neural approximation of value functions in high dimensions

Successful computation of MFG equilibria using coupled neural networks

Improved convergence in solving high-dimensional control problems

Abstract

This paper first introduces a method to approximate the value function of high-dimensional optimal control by neural networks. Based on the established relationship between Pontryagin's maximum principle (PMP) and the value function of the optimal control problem, which is characterized as being the unique solution to an associated Hamilton-Jacobi-Bellman (HJB) equation, we propose an approach that begins by using neural networks to provide a first rough estimate of the value function, which serves as initialization for solving the two point boundary value problem in the PMP and, as a result, generates reliable data. To train the neural network we define a loss function that takes into account this dataset and also penalizes deviations from the HJB equation. In the second part, we address the computation of equilibria in first-order Mean Field Game (MFG) problems by integrating our method with the fictitious play algorithm. These equilibria are characterized by a coupled system of a first-order HJB equation and a continuity equation. To approximate the solution to the continuity equation, we introduce a second neural network that learns the flow map transporting the initial distribution of agents. This network is trained on data generated by solving the underlying ODEs for a batch of initial conditions sampled from the initial distribution of agents. By combining this flow approximation, the previously described method for approximating the value function, and the fictitious play algorithm, we obtain an effective method to tackle high-dimensional deterministic MFGs.