Online Testing of Subgroup Treatment Effects Based on Value Difference

TL;DR

This paper introduces SUBTLE, a sequential testing method for subgroup treatment effects in online A/B testing, addressing false positives and heterogeneity, with proven high detection power and early stopping capabilities.

Contribution

The paper proposes a novel sequential test called SUBTLE that detects beneficial subgroups in online experiments while controlling false positives and allowing continuous monitoring.

Findings

SUBTLE maintains high detection power and controls type I error at any time.

It is more robust to noise covariates than traditional methods.

SUBTLE enables early stopping in experiments, saving resources.

Abstract

Online A/B testing plays a critical role in the high-tech industry to guide product development and accelerate innovation. It performs a null hypothesis statistical test to determine which variant is better. However, a typical A/B test presents two problems: (i) a fixed-horizon framework inflates the false-positive errors under continuous monitoring; (ii) the homogeneous effects assumption fails to identify a subgroup with a beneficial treatment effect. In this paper, we propose a sequential test for subgroup treatment effects based on value difference, named SUBTLE, to address these two problems simultaneously. The SUBTLE allows the experimenters to "peek" at the results during the experiment without harming the statistical guarantees. It assumes heterogeneous treatment effects and aims to test if some subgroup of the population will benefit from the investigative treatment. If the testing result indicates the existence of such a subgroup, a subgroup will be identified using a readily available estimated optimal treatment rule. We examine the empirical performance of our proposed test on both simulations and a real dataset. The results show that the SUBTLE has high detection power with controlled type I error at any time, is more robust to noise covariates, and can achieve early stopping compared with the corresponding fixed-horizon test.