Early Detection of Long Term Evaluation Criteria in Online Controlled Experiments

TL;DR

This paper introduces a bootstrap hypothesis testing method for early detection of long-term effects in online experiments, enabling faster and more accurate evaluation of metrics that matter most over time.

Contribution

It proposes a novel, efficient methodology for early detection of long-term differences in online experiments using bootstrap hypothesis testing.

Findings

Method accelerates detection of long-term effects

Applicable to online advertising experiments

Focuses on metrics with long-term impact

Abstract

A common dilemma encountered by many upon implementing an optimization method or experiment, whether it be a reinforcement learning algorithm, or A/B testing, is deciding on what metric to optimize for. Very often short-term metrics, which are easier to measure are chosen over long term metrics which have undesirable time considerations and often a more complex calculation. In this paper, we argue the importance of choosing a metrics that focuses on long term effects. With this comes the necessity in the ability to measure significant differences between groups relatively early. We present here an efficient methodology for early detection of lifetime differences between groups based on bootstrap hypothesis testing of the lifetime forecast of the response. We present an application of this method in the domain of online advertising and we argue that approach not only allows one to focus on the ultimate metric of importance but also provides a means of accelerating the testing period.