Predicting Long Term Sequential Policy Value Using Softer Surrogates

TL;DR

This paper introduces a method to predict the long-term outcomes of policies in healthcare by combining short-term on-policy data with long-term historical data, addressing challenges in off-policy evaluation with novel actions.

Contribution

It proposes a novel surrogacy-based approach for long-term policy value prediction using short-term data, applicable when new actions are introduced.

Findings

Accurate long-term policy value predictions with only 10% of data observed.

Method effective in simulated healthcare scenarios like HIV and sepsis management.

Finite sample analysis supports estimator robustness.

Abstract

Off-policy policy evaluation (OPE) estimates the outcome of a new policy using historical data collected from a different policy. However, existing OPE methods cannot handle cases when the new policy introduces novel actions. This issue commonly occurs in real-world domains, like healthcare, as new drugs and treatments are continuously developed. Novel actions necessitate on-policy data collection, which can be burdensome and expensive if the outcome of interest takes a substantial amount of time to observe--for example, in multi-year clinical trials. This raises a key question of how to predict the long-term outcome of a policy after only observing its short-term effects? Though in general this problem is intractable, under some surrogacy conditions, the short-term on-policy data can be combined with the long-term historical data to make accurate predictions about the new policy's long-term value. In two simulated healthcare examples--HIV and sepsis management--we show that our estimators can provide accurate predictions about the policy value only after observing 10\% of the full horizon data. We also provide finite sample analysis of our doubly robust estimators.