Policy Evaluation in Decentralized POMDPs with Belief Sharing

TL;DR

This paper introduces a decentralized belief sharing method for cooperative policy evaluation in multi-agent systems with noisy observations, enabling agents to approximate centralized performance through local interactions.

Contribution

It proposes a novel decentralized belief formation strategy that facilitates information diffusion and parameter convergence in multi-agent POMDPs with limited communication.

Findings

Belief sharing improves policy evaluation accuracy.

Agents' parameters stay close to centralized baseline.

Method effective in multi-sensor target tracking simulations.

Abstract

Most works on multi-agent reinforcement learning focus on scenarios where the state of the environment is fully observable. In this work, we consider a cooperative policy evaluation task in which agents are not assumed to observe the environment state directly. Instead, agents can only have access to noisy observations and to belief vectors. It is well-known that finding global posterior distributions under multi-agent settings is generally NP-hard. As a remedy, we propose a fully decentralized belief forming strategy that relies on individual updates and on localized interactions over a communication network. In addition to the exchange of the beliefs, agents exploit the communication network by exchanging value function parameter estimates as well. We analytically show that the proposed strategy allows information to diffuse over the network, which in turn allows the agents' parameters to have a bounded difference with a centralized baseline. A multi-sensor target tracking application is considered in the simulations.