Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability

TL;DR

This paper introduces Multi-Environment POMDPs (ME-POMDPs) to handle discrete model uncertainty across multiple domains, providing algorithms for robust policy computation under partial observability and model disagreement.

Contribution

It generalizes ME-POMDPs to adversarial-belief POMDPs and offers reduction techniques and algorithms for robust policy computation in multi-environment settings.

Findings

Algorithms for exact and approximate robust policy computation.

Extension of POMDP benchmarks to multi-environment scenarios.

Demonstration of policies on standard benchmarks.

Abstract

Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty. ME-POMDPs represent a finite set of POMDPs that share the same state, action, and observation spaces, but may arbitrarily vary in their transition, observation, and reward models. Such models arise, for instance, when multiple domain experts disagree on how to model a problem. The goal is to find a single policy that is robust against any choice of POMDP within the set, i.e., a policy that maximizes the worst-case reward across all POMDPs. We generalize and expand on existing work in the following way. First, we show that ME-POMDPs can be generalized to POMDPs with sets of initial beliefs, which we call adversarial-belief POMDPs (AB-POMDPs). Second, we show that any arbitrary ME-POMDP can be reduced to a ME-POMDP that only varies in its transition and reward functions or only in its observation and reward functions, while preserving (optimal) policies. We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs, and thus ME-POMDPs. We demonstrate that we can compute policies for standard POMDP benchmarks extended to the multi-environment setting.