A Certainty Equivalence Result in Team-Optimal Control of Mean-Field Coupled Markov Chains

TL;DR

This paper presents a decentralized control approach for large populations of Markov decision processes with mean-field coupling, providing a sub-optimal solution that converges to the optimal mean-field solution as the number of processes grows.

Contribution

It introduces a sub-optimal decentralized control method for mean-field coupled Markov chains that converges to the optimal solution with a quantifiable rate.

Findings

The proposed solution converges at a rate proportional to the square root of the inverse of the number of processes.

A combinatorial optimization problem is formulated to achieve the same convergence rate.

The method applies under mild conditions and does not depend on the total number of processes.

Abstract

This paper studies a large number of homogeneous Markov decision processes where the transition probabilities and costs are coupled in the empirical distribution of states (also called mean-field). The state of each process is not known to others, which means that the information structure is fully decentralized. The objective is to minimize the average cost, defined as the empirical mean of individual costs, for which a sub-optimal solution is proposed. This solution does not depend on the number of processes, yet it converges to the optimal solution of the so-called mean-field sharing as the number of processes tends to infinity. Under some mild conditions, it is shown that the convergence rate of the proposed decentralized solution is proportional to the square root of the inverse of the number of processes. Finding this sub-optimal solution involves a non-smooth non-convex optimization problem over an uncountable set, in general. To overcome this drawback, a combinatorial optimization problem is introduced that achieves the same rate of convergence.