Value Iteration Algorithm for Mean-field Games

TL;DR

This paper introduces a value iteration algorithm using Q-functions to compute mean-field equilibria in discrete-time mean field games, applicable to both discounted and average cost criteria, and proves its convergence.

Contribution

It presents the first value iteration algorithm for mean-field games that guarantees convergence to equilibrium using Q-functions.

Findings

Algorithm converges to mean-field equilibrium fixed point.

Applicable to both discounted and average cost criteria.

Provides a constructive method for equilibrium computation.

Abstract

In the literature, existence of mean-field equilibria has been established for discrete-time mean field games under both the discounted cost and the average cost optimality criteria. In this paper, we provide a value iteration algorithm to compute mean-field equilibrium for both the discounted cost and the average cost criteria, whose existence proved previously. We establish that the value iteration algorithm converges to the fixed point of a mean-field equilibrium operator. Then, using this fixed point, we construct a mean-field equilibrium. In our value iteration algorithm, we use $Q$-functions instead of value functions.