An Online Sequential Test for Qualitative Treatment Effects

TL;DR

This paper introduces a scalable online testing method to detect qualitative treatment effects in A/B testing, with adaptive features and theoretical guarantees, enhancing the ability to identify specific circumstances where new treatments outperform existing ones.

Contribution

It develops a novel online testing procedure with adaptive randomization, sequential monitoring, and online updating, ensuring controlled type-I error and improved detection of qualitative effects.

Findings

The proposed test controls type-I error effectively.

The method demonstrates strong finite sample performance.

Theoretical properties are rigorously validated.

Abstract

Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one. However, it is also critically important to detect qualitative treatment effects such that the new one may significantly outperform the existing one only under some specific circumstances. The aim of this paper is to develop a powerful testing procedure to efficiently detect such qualitative treatment effects. We propose a scalable online updating algorithm to implement our test procedure. It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control. We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justification of an efficient bootstrap method. Extensive empirical studies are conducted to examine the finite sample performance of our test procedure.