Targeting for long-term outcomes

TL;DR

This paper develops a method to optimize long-term intervention targeting by imputing missing long-term outcomes using surrogacy and policy learning, validated through large-scale experiments at The Boston Globe.

Contribution

It introduces a doubly-robust approach for policy optimization with imputed long-term outcomes, relaxing conditions for valid policy evaluation and demonstrating practical success in real-world experiments.

Findings

Policies based on imputed outcomes perform as well as those based on true long-term outcomes.

The approach outperforms short-term proxy-based policies in long-term revenue maximization.

Implemented policies generated a net revenue increase of $4-5 million over three years.

Abstract

Decision makers often want to target interventions so as to maximize an outcome that is observed only in the long-term. This typically requires delaying decisions until the outcome is observed or relying on simple short-term proxies for the long-term outcome. Here we build on the statistical surrogacy and policy learning literatures to impute the missing long-term outcomes and then approximate the optimal targeting policy on the imputed outcomes via a doubly-robust approach. We first show that conditions for the validity of average treatment effect estimation with imputed outcomes are also sufficient for valid policy evaluation and optimization; furthermore, these conditions can be somewhat relaxed for policy optimization. We apply our approach in two large-scale proactive churn management experiments at The Boston Globe by targeting optimal discounts to its digital subscribers with the aim of maximizing long-term revenue. Using the first experiment, we evaluate this approach empirically by comparing the policy learned using imputed outcomes with a policy learned on the ground-truth, long-term outcomes. The performance of these two policies is statistically indistinguishable, and we rule out large losses from relying on surrogates. Our approach also outperforms a policy learned on short-term proxies for the long-term outcome. In a second field experiment, we implement the optimal targeting policy with additional randomized exploration, which allows us to update the optimal policy for future subscribers. Over three years, our approach had a net-positive revenue impact in the range of $4-5 million compared to the status quo.