Anytime Planning for Decentralized POMDPs using Expectation Maximization

TL;DR

This paper introduces a novel approach for infinite-horizon decentralized POMDPs by framing the policy optimization as inference in dynamic Bayesian networks and employing Expectation Maximization, enabling richer representations and improved performance.

Contribution

It presents a new class of algorithms that recast infinite-horizon DEC-POMDP optimization as inference in DBNs, utilizing EM for joint policy optimization, and demonstrates superior results on benchmarks.

Findings

EM-based approach outperforms existing solvers

Supports richer state and action representations

Enables inference techniques in complex DEC-POMDPs

Abstract

Decentralized POMDPs provide an expressive framework for multi-agent sequential decision making. While fnite-horizon DECPOMDPs have enjoyed signifcant success, progress remains slow for the infnite-horizon case mainly due to the inherent complexity of optimizing stochastic controllers representing agent policies. We present a promising new class of algorithms for the infnite-horizon case, which recasts the optimization problem as inference in a mixture of DBNs. An attractive feature of this approach is the straightforward adoption of existing inference techniques in DBNs for solving DEC-POMDPs and supporting richer representations such as factored or continuous states and actions. We also derive the Expectation Maximization (EM) algorithm to optimize the joint policy represented as DBNs. Experiments on benchmark domains show that EM compares favorably against the state-of-the-art solvers.