Causal Temporal Reasoning for Markov Decision Processes

TL;DR

This paper introduces PCFTL, a novel probabilistic temporal logic that incorporates causal reasoning for MDPs, enabling analysis of interventional and counterfactual scenarios in reinforcement learning.

Contribution

It presents the first logic to include causal operators for MDP verification, bridging probabilistic temporal logic with structural causal models for counterfactual reasoning.

Findings

Enables reasoning about 'what-if' scenarios in MDPs

Unified framework for interventional and counterfactual probabilities

Validated on safe reinforcement learning grid-world benchmarks

Abstract

We introduce $\textit{PCFTL (Probabilistic CounterFactual Temporal Logic)}$, a new probabilistic temporal logic for the verification of Markov Decision Processes (MDP). PCFTL is the first to include operators for causal reasoning, allowing us to express interventional and counterfactual queries. Given a path formula $\phi$, an interventional property is concerned with the satisfaction probability of $\phi$ if we apply a particular change $I$ to the MDP (e.g., switching to a different policy); a counterfactual allows us to compute, given an observed MDP path $\tau$, what the outcome of $\phi$ would have been had we applied $I$ in the past. For its ability to reason about \textit{what-if} scenarios involving different configurations of the MDP, our approach represents a departure from existing probabilistic temporal logics that can only reason about a fixed system configuration. From a syntactic viewpoint, we introduce a generalized counterfactual operator that subsumes both interventional and counterfactual probabilities as well as the traditional probabilistic operator found in e.g., PCTL. From a semantics viewpoint, our logic is interpreted over a structural causal model translation of the MDP, which gives us a representation amenable to counterfactual reasoning. We evaluate PCFTL in the context of safe reinforcement learning using a benchmark of grid-world models.