Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

TL;DR

This paper introduces CF-GPS, a counterfactually-guided policy search method that uses structural causal models to improve policy learning from logged data in POMDPs, addressing biases in model-based RL.

Contribution

The paper presents CF-GPS, a novel algorithm that leverages counterfactual reasoning with structural causal models to enhance policy search from off-policy data.

Findings

CF-GPS improves policy evaluation and search in complex environments.

It outperforms vanilla model-based RL algorithms on a grid-world task.

CF-GPS generalizes Guided Policy Search and relates to reparameterization methods.

Abstract

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.