An Evaluation Framework for Personalization Strategy Experiment Designs

TL;DR

This paper introduces an evaluation framework for designing online experiments on personalization strategies, emphasizing proper user allocation to improve detection power and scalability.

Contribution

It proposes a new framework and simple rules to help experimenters choose optimal setup configurations for higher test power in personalization experiments.

Findings

Framework enables quick comparison of experiment setups.

Proper user allocation improves detection probability.

Supports scaling of personalization experiments.

Abstract

Online Controlled Experiments (OCEs) are the gold standard in evaluating the effectiveness of changes to websites. An important type of OCE evaluates different personalization strategies, which present challenges in low test power and lack of full control in group assignment. We argue that getting the right experiment setup -- the allocation of users to treatment/analysis groups -- should take precedence of post-hoc variance reduction techniques in order to enable the scaling of the number of experiments. We present an evaluation framework that, along with a few simple rule of thumbs, allow experimenters to quickly compare which experiment setup will lead to the highest probability of detecting a treatment effect under their particular circumstance.