PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data

TL;DR

This paper introduces two new methods for constructing valid confidence intervals in off-policy evaluation of reinforcement learning policies, effectively leveraging auxiliary data while providing reliable uncertainty quantification in high-stakes domains.

Contribution

It proposes a conformal prediction approach for high-dimensional state MDPs and a doubly robust inference method for average policy performance, enabling principled uncertainty estimates with augmented data.

Findings

Methods produce intervals that reliably cover true policy values.

Approaches outperform existing methods in diverse simulators and healthcare data.

Validated on MIMIC-IV dataset and multiple simulation environments.

Abstract

Off-policy evaluation (OPE) methods aim to estimate the value of a new reinforcement learning (RL) policy prior to deployment. Recent advances have shown that leveraging auxiliary datasets, such as those synthesized by generative models, can improve the accuracy of these value estimates. Unfortunately, such auxiliary datasets may also be biased, and existing methods for using data augmentation for OPE in RL lack principled uncertainty quantification. In high stakes settings like healthcare, reliable uncertainty estimates are important for comparing policy value estimates. In this work, we propose two approaches to construct valid confidence intervals for OPE when using data augmentation. The first provides a confidence interval over the policy performance conditioned on a particular initial state $V^{\pi}(s_0)$-- such intervals are particularly important for human-centered applications. To do so we introduce a new conformal prediction method for high dimensional state MDPs. Second, we consider the more common task of estimating the average policy performance over many initial states; to do so we draw on ideas from doubly robust estimation and prediction powered inference. Across simulators spanning robotics, healthcare and inventory management, and a real healthcare dataset from MIMIC-IV, we find that our methods can use augmented data and still consistently produce intervals that cover the ground truth values, unlike previously proposed methods.