A nonparametric sequential test for online randomized experiments

TL;DR

This paper introduces a nonparametric sequential testing method for online randomized experiments that controls type I error, handles complex metrics, and is robust to distribution misspecification, enabling quick and reliable inference.

Contribution

It presents a novel bootstrap-based mixture sequential probability ratio test that does not require distributional assumptions and effectively manages continuous monitoring in online experiments.

Findings

Controls type I error at any time

Demonstrates good statistical power

Robust to distribution misspecification

Abstract

We propose a nonparametric sequential test that aims to address two practical problems pertinent to online randomized experiments: (i) how to do a hypothesis test for complex metrics; (ii) how to prevent type $1$ error inflation under continuous monitoring. The proposed test does not require knowledge of the underlying probability distribution generating the data. We use the bootstrap to estimate the likelihood for blocks of data followed by mixture sequential probability ratio test. We validate this procedure on data from a major online e-commerce website. We show that the proposed test controls type $1$ error at any time, has good power, is robust to misspecification in the distribution generating the data, and allows quick inference in online randomized experiments.