On Bellman's Optimality Principle for zs-POSGs

TL;DR

This paper extends Bellman's optimality principle to infinite horizon 2-player zero-sum partially observable stochastic games by transforming them into occupancy Markov games and applying a Lipschitz-continuous value function approach, enabling finite-time epsilon-Nash equilibria.

Contribution

It introduces a novel approach to apply Bellman's principle to zs-POSGs via occupancy states and develops a HSVI-based algorithm with proven convergence guarantees.

Findings

The method computes epsilon-Nash equilibria in finite time.

Occupancy space Lipschitz continuity enables value iteration.

Transformation to occupancy Markov games simplifies analysis.

Abstract

Many non-trivial sequential decision-making problems are efficiently solved by relying on Bellman's optimality principle, i.e., exploiting the fact that sub-problems are nested recursively within the original problem. Here we show how it can apply to (infinite horizon) 2-player zero-sum partially observable stochastic games (zs-POSGs) by (i) taking a central planner's viewpoint, which can only reason on a sufficient statistic called occupancy state, and (ii) turning such problems into zero-sum occupancy Markov games (zs-OMGs). Then, exploiting the Lipschitz-continuity of the value function in occupancy space, one can derive a version of the HSVI algorithm (Heuristic Search Value Iteration) that provably finds an $\epsilon$-Nash equilibrium in finite time.