Approximating the Uniform Value in Hidden Stochastic Games with Doeblin Conditions

TL;DR

This paper proves the existence of the uniform value and provides an approximation algorithm for hidden stochastic games under Doeblin conditions, extending results to partially observable settings.

Contribution

It introduces Doeblin conditions to hidden stochastic games, ensuring uniform value existence and enabling approximation algorithms, a novel approach in this context.

Findings

Uniform value exists under Doeblin conditions.

An algorithm for approximating the uniform value is provided.

Ergodicity does not imply Doeblin condition in hidden settings.

Abstract

In \emph{zero-sum two-player hidden stochastic games}, players observe partial information about the state. We address: $(i)$ the existence of the \emph{uniform value}, i.e., a limiting average payoff that both players can guarantee for sufficiently long durations, and $(ii)$ the existence of an algorithm to approximate it. Previous work shows that, in the general case, the uniform value may fail to exist, and, even when it does, there need not exist an algorithm to compute or approximate it. Therefore, we consider the \emph{Doeblin condition} in hidden stochastic games, requiring that, after a sufficiently long time, the posterior beliefs have a uniformly positive probability of resetting to one of finitely many neighborhoods in the belief space. We prove the existence of the uniform value and provide an algorithm to approximate it. We identify sufficient conditions, namely \emph{ergodicity} in the blind setting (when the signal is uninformative) and \emph{primitivity} in the hidden setting (when there are multiple signals). Moreover, we show that, in the hidden setting, ergodicity does not guarantee the Doeblin condition. Our results are new even for the one-player setting, i.e., partially observable Markov decision processes.