Multi-Environment MDPs with Prior and Universal Semantics

TL;DR

This paper investigates multi-environment MDPs with prior and universal semantics, providing new algorithms for value approximation, clarifying their relationship, and connecting prior-MEMDPs to a tractable subclass of POMDPs.

Contribution

It introduces algorithms for approximating values in MEMDPs with prior semantics, clarifies the relation between prior and universal semantics, and links prior-MEMDPs to a tractable subclass of POMDPs.

Findings

Prior value approximation is space-efficient and decidable in PSPACE.

Universal value equals the infimum of prior values over beliefs.

Prior-MEMDPs form a tractable subclass of POMDPs with non-increasing belief entropy.

Abstract

Multiple-environment Markov decision processes (MEMDPs) equip an MDP with several probabilistic transition functions (one per possible environment) so that the state is observable but the environment is not. Previous work studies two semantics: (i) the universal semantics, where an adversary picks the environment; and (ii) the prior semantics, where the environment is drawn once before execution from a fixed distribution. We clarify the relation between these semantics. For parity objectives, we show that the qualitative questions, i.e. value one, coincide, and we develop a new algorithm for the general value of MEMDP with prior semantics. In particular, we show that the prior value of an MEMDP with a parity objective can be approximated to any precision with a space efficient algorithm; equivalently, the associated gap problem is decidable in PSPACE when probabilities are given in unary (and in EXPSPACE otherwise). We then prove that the universal value equals the infimum of prior values over all beliefs. This yields a new algorithm for the universal gap problem with the same complexity (PSPACE for unary probabilities, EXPSPACE in general), improving on earlier doubly-exponential-space procedures. Finally, we observe that MEMDPs under the prior semantics form an important tractable subclass of POMDPs: our algorithms exploit the fact that belief entropy never increases, and we establish that any POMDP with this property reduces effectively to a prior-MEMDP, showing that prior-MEMDPs capture a broad and practically relevant subclass of POMDPs.