Zero to Hero: Exploiting Null Effects to Achieve Variance Reduction in Experiments with One-sided Triggering

TL;DR

This paper introduces an unbiased variance reduction method for online experiments with limited triggering data, improving precision in treatment effect estimation by leveraging covariate adjustments and assumptions.

Contribution

It proposes a novel unbiased estimator for intent-to-treat effects that reduces variance using covariate adjustment when triggering status is only observed in treated subjects.

Findings

Estimator remains unbiased under certain assumptions

Achieves variance reduction comparable to full triggering data scenarios

Outperforms traditional trigger-dilute analysis in simulations and real experiments

Abstract

In online experiments where the intervention is only exposed, or "triggered", for a small subset of the population, it is critical to use variance reduction techniques to estimate treatment effects with sufficient precision to inform business decisions. Trigger-dilute analysis is often used in these situations, and reduces the sampling variance of overall intent-to-treat (ITT) effects by an order of magnitude equal to the inverse of the triggering rate; for example, a triggering rate of $5\%$ corresponds to roughly a $20x$ reduction in variance. To apply trigger-dilute analysis, one needs to know experimental subjects' triggering counterfactual statuses, i.e., the counterfactual behavior of subjects under both treatment and control conditions. In this paper, we propose an unbiased ITT estimator with reduced variance applicable for experiments where the triggering counterfactual status is only observed in the treatment group. Our method is based on the efficiency augmentation idea of CUPED and draws upon identification frameworks from the principal stratification and instrumental variables literature. The unbiasedness of our estimation approach relies on a testable assumption that the augmentation term used for covariate adjustment equals zero in expectation. Unlike traditional covariate adjustment or principal score modeling approaches, our estimator can incorporate both pre-experiment and in-experiment observations. We demonstrate through a real-world experiment and simulations that our estimator can remain unbiased and achieve precision improvements as large as if triggering status were fully observed, and in some cases can even outperform trigger-dilute analysis.