Counterfactual equivalence for POMDPs, and underlying deterministic environments

TL;DR

This paper introduces the concepts of equivalence and counterfactual equivalence in POMDPs, demonstrating that any POMDP can be represented as a deterministic environment with initial state uncertainty, enhancing understanding of uncertainty and learning.

Contribution

It formalizes counterfactual equivalence in POMDPs and proves that any POMDP is equivalent to a deterministic one with initial state uncertainty, providing new insights into their structure.

Findings

Any POMDP is counterfactually equivalent to a deterministic environment.

Uncertainty in POMDPs can be concentrated into the initial state.

This equivalence aids in understanding POMDP information and learning processes.

Abstract

Partially Observable Markov Decision Processes (POMDPs) are rich environments often used in machine learning. But the issue of information and causal structures in POMDPs has been relatively little studied. This paper presents the concepts of equivalent and counterfactually equivalent POMDPs, where agents cannot distinguish which environment they are in though any observations and actions. It shows that any POMDP is counterfactually equivalent, for any finite number of turns, to a deterministic POMDP with all uncertainty concentrated into the initial state. This allows a better understanding of POMDP uncertainty, information, and learning.