Optimally Solving Simultaneous-Move Dec-POMDPs: The Sequential Central Planning Approach

TL;DR

This paper introduces a scalable sequential-move centralized training approach for decentralized decision-making in multi-agent systems, improving efficiency and enabling the use of single-agent algorithms while maintaining near-optimal performance.

Contribution

It proposes a novel sequential-move paradigm that reduces complexity and extends Bellman's principle to multi-agent partially observable settings, facilitating more scalable solutions.

Findings

Outperforms existing simultaneous-move solvers in experiments

Reduces backup operator complexity from double exponential to polynomial

Enables use of single-agent algorithms like SARSA with guarantees

Abstract

The centralized training for decentralized execution paradigm emerged as the state-of-the-art approach to $\epsilon$-optimally solving decentralized partially observable Markov decision processes. However, scalability remains a significant issue. This paper presents a novel and more scalable alternative, namely the sequential-move centralized training for decentralized execution. This paradigm further pushes the applicability of the Bellman's principle of optimality, raising three new properties. First, it allows a central planner to reason upon sufficient sequential-move statistics instead of prior simultaneous-move ones. Next, it proves that $\epsilon$-optimal value functions are piecewise linear and convex in such sufficient sequential-move statistics. Finally, it drops the complexity of the backup operators from double exponential to polynomial at the expense of longer planning horizons. Besides, it makes it easy to use single-agent methods, e.g., SARSA algorithm enhanced with these findings, while still preserving convergence guarantees. Experiments on two- as well as many-agent domains from the literature against $\epsilon$-optimal simultaneous-move solvers confirm the superiority of our novel approach. This paradigm opens the door for efficient planning and reinforcement learning methods for multi-agent systems.