Cooperative Multi-Agent Constrained POMDPs: Strong Duality and Primal-Dual Reinforcement Learning with Approximate Information States

TL;DR

This paper establishes strong duality for decentralized constrained POMDPs with asymmetric information, and develops a primal-dual reinforcement learning framework using neural networks for multi-agent systems.

Contribution

It proves strong duality under certain conditions and introduces a novel primal-dual MARL method with approximate information-states that are independent of Lagrange multipliers.

Findings

Strong duality holds for infinite-horizon discounted costs in certain constrained POMDPs.

The proposed primal-dual MARL framework effectively learns policies with neural network approximators.

Approximate information-states enable adaptation without redefining representations during learning.

Abstract

We study the problem of decentralized constrained POMDPs in a team-setting where the multiple non-strategic agents have asymmetric information. Strong duality is established for the setting of infinite-horizon expected total discounted costs when the observations lie in a countable space, the actions are chosen from a finite space, and the immediate cost functions are bounded. Following this, connections with the common-information and approximate information-state approaches are established. The approximate information-states are characterized independent of the Lagrange-multipliers vector so that adaptations of the multiplier (during learning) will not necessitate new representations. Finally, a primal-dual multi-agent reinforcement learning (MARL) framework based on centralized training distributed execution (CTDE) and three time-scale stochastic approximation is developed with the aid of recurrent and feedforward neural-networks as function-approximators.