Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models

TL;DR

This paper presents a novel off-policy evaluation method using Gumbel-Max Structural Causal Models to identify episodes with significant reward differences, aiding debugging in high-stakes RL applications like healthcare.

Contribution

It introduces a new class of SCMs for counterfactual trajectory generation in POMDPs, enabling episode-level analysis of policy differences for safer RL deployment.

Findings

Effective identification of episodes with large reward discrepancies

Demonstrated utility in a synthetic sepsis management environment

Facilitates targeted review by domain experts

Abstract

We introduce an off-policy evaluation procedure for highlighting episodes where applying a reinforcement learned (RL) policy is likely to have produced a substantially different outcome than the observed policy. In particular, we introduce a class of structural causal models (SCMs) for generating counterfactual trajectories in finite partially observable Markov Decision Processes (POMDPs). We see this as a useful procedure for off-policy "debugging" in high-risk settings (e.g., healthcare); by decomposing the expected difference in reward between the RL and observed policy into specific episodes, we can identify episodes where the counterfactual difference in reward is most dramatic. This in turn can be used to facilitate review of specific episodes by domain experts. We demonstrate the utility of this procedure with a synthetic environment of sepsis management.