Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimation

TL;DR

This paper extends decentralized mean-field game algorithms to include function approximation, enabling agents to estimate and communicate about the global mean field, thus improving policy learning in larger, more complex environments.

Contribution

It introduces function approximation into networked mean-field game algorithms and develops methods for local empirical mean-field estimation through inter-agent communication.

Findings

Communication improves agents' policy performance.

Agents can accurately estimate the mean field locally.

Algorithms outperform both independent and centralized baselines.

Abstract

Recent algorithms allow decentralised agents, possibly connected via a communication network, to learn equilibria in mean-field games from a non-episodic run of the empirical system. However, these algorithms are for tabular settings: this computationally limits the size of agents' observation space, meaning the algorithms cannot handle anything but small state spaces, nor generalise beyond policies depending only on the agent's local state to so-called 'population-dependent' policies. We address this limitation by introducing function approximation to the existing setting, drawing on the Munchausen Online Mirror Descent method that has previously been employed only in finite-horizon, episodic, centralised settings. While this permits us to include the mean field in the observation for players' policies, it is unrealistic to assume decentralised agents have access to this global information: we therefore also provide new algorithms allowing agents to locally estimate the global empirical distribution, and to improve this estimate via inter-agent communication. We prove theoretically that exchanging policy information helps networked agents outperform both independent and even centralised agents in function-approximation settings. Our experiments demonstrate this happening empirically, and show that the communication network allows decentralised agents to estimate the mean field for population-dependent policies.