Optimizing Memory-Bounded Controllers for Decentralized POMDPs

TL;DR

This paper introduces a memory-bounded nonlinear optimization method for decentralized POMDPs, producing higher quality policies than existing approaches by leveraging stochastic controllers and shared randomness.

Contribution

It formulates decentralized POMDP policy optimization as a nonlinear program, improving solution quality and incorporating correlation devices for enhanced performance.

Findings

Higher quality controllers than state-of-the-art methods

Effective use of nonlinear optimization techniques

Shared randomness improves solution quality with limited overhead

Abstract

We present a memory-bounded optimization approach for solving infinite-horizon decentralized POMDPs. Policies for each agent are represented by stochastic finite state controllers. We formulate the problem of optimizing these policies as a nonlinear program, leveraging powerful existing nonlinear optimization techniques for solving the problem. While existing solvers only guarantee locally optimal solutions, we show that our formulation produces higher quality controllers than the state-of-the-art approach. We also incorporate a shared source of randomness in the form of a correlation device to further increase solution quality with only a limited increase in space and time. Our experimental results show that nonlinear optimization can be used to provide high quality, concise solutions to decentralized decision problems under uncertainty.