Should I Stop or Should I Go: Early Stopping with Heterogeneous Populations

TL;DR

This paper introduces CLASH, a novel causal machine learning method for early stopping in experiments involving heterogeneous populations, effectively identifying harm to minority groups in clinical trials and A/B testing.

Contribution

It develops the first broadly applicable method for heterogeneous early stopping, addressing limitations of existing aggregate-based approaches.

Findings

CLASH outperforms existing methods in simulations and real data.

It effectively detects harm to minority groups during experiments.

CLASH improves early stopping decisions in clinical trials and A/B tests.

Abstract

Randomized experiments often need to be stopped prematurely due to the treatment having an unintended harmful effect. Existing methods that determine when to stop an experiment early are typically applied to the data in aggregate and do not account for treatment effect heterogeneity. In this paper, we study the early stopping of experiments for harm on heterogeneous populations. We first establish that current methods often fail to stop experiments when the treatment harms a minority group of participants. We then use causal machine learning to develop CLASH, the first broadly-applicable method for heterogeneous early stopping. We demonstrate CLASH's performance on simulated and real data and show that it yields effective early stopping for both clinical trials and A/B tests.