Choosing a Proxy Metric from Past Experiments

TL;DR

This paper presents a statistical framework for selecting and constructing optimal proxy metrics from past experiments to better estimate long-term effects in short-term, noisy, randomized experiments, adapting to sample size and noise levels.

Contribution

It introduces a novel portfolio optimization approach to define and construct proxy metrics based on historical data, improving decision-making in experimental settings.

Findings

Constructed proxy metrics outperform baselines in real experiments.

Optimal proxy metrics depend on experiment sample size and noise levels.

Framework effectively denoises long-term effects using historical data.

Abstract

In many randomized experiments, the treatment effect of the long-term metric (i.e. the primary outcome of interest) is often difficult or infeasible to measure. Such long-term metrics are often slow to react to changes and sufficiently noisy they are challenging to faithfully estimate in short-horizon experiments. A common alternative is to measure several short-term proxy metrics in the hope they closely track the long-term metric -- so they can be used to effectively guide decision-making in the near-term. We introduce a new statistical framework to both define and construct an optimal proxy metric for use in a homogeneous population of randomized experiments. Our procedure first reduces the construction of an optimal proxy metric in a given experiment to a portfolio optimization problem which depends on the true latent treatment effects and noise level of experiment under consideration. We then denoise the observed treatment effects of the long-term metric and a set of proxies in a historical corpus of randomized experiments to extract estimates of the latent treatment effects for use in the optimization problem. One key insight derived from our approach is that the optimal proxy metric for a given experiment is not apriori fixed; rather it should depend on the sample size (or effective noise level) of the randomized experiment for which it is deployed. To instantiate and evaluate our framework, we employ our methodology in a large corpus of randomized experiments from an industrial recommendation system and construct proxy metrics that perform favorably relative to several baselines.