Bridging the Gap between Partially Observable Stochastic Games and Sparse POMDP Methods

TL;DR

This paper introduces a unified framework combining POMDP techniques and game-theoretic methods to efficiently solve large-scale partially observable stochastic games, enhancing autonomous multi-agent decision-making.

Contribution

It presents a novel approach that unifies belief approximation and equilibrium search, providing a theoretical foundation and empirical validation for scalable POSG solutions.

Findings

Effective belief approximation bounds the error in POSG planning.

The approach enables handling very large state spaces in multi-agent scenarios.

Empirical results demonstrate improved planning in complex environments.

Abstract

Many real-world decision problems involve the interaction of multiple self-interested agents with limited sensing ability. The partially observable stochastic game (POSG) provides a mathematical framework for modeling these problems, however solving a POSG requires difficult reasoning over two critical factors: (1) information revealed by partial observations and (2) decisions other agents make. In the single agent case, partially observable Markov decision process (POMDP) planning can efficiently address partial observability with particle filtering. In the multi-agent case, extensive form game solution methods account for other agent's decisions, but preclude belief approximation. We propose a unifying framework that combines POMDP-inspired state distribution approximation and game-theoretic equilibrium search on information sets. This paper lays a theoretical foundation for the approach by bounding errors due to belief approximation, and empirically demonstrates effectiveness with a numerical example. The new approach enables planning in POSGs with very large state spaces, paving the way for reliable autonomous interaction in real-world physical environments and complementing multi-agent reinforcement learning.