Improving precision of A/B experiments using trigger intensity

TL;DR

This paper introduces a sampling-based evaluation method to improve the precision of A/B experiments by reducing the cost of trigger detection, achieving significant standard error reduction with minimal bias.

Contribution

It proposes a novel sampling approach for trigger observation estimation in A/B tests, balancing cost and bias, with theoretical analysis and empirical validation.

Findings

Bias decreases inversely with sample size

Partial knowledge reduces standard error by 36.48%

Sampling as low as 0.1% of trigger observations suffices

Abstract

In industry, online randomized controlled experiment (a.k.a. A/B experiment) is a standard approach to measure the impact of a causal change. These experiments have small treatment effect to reduce the potential blast radius. As a result, these experiments often lack statistical significance due to low signal-to-noise ratio. A standard approach for improving the precision (or reducing the standard error) focuses only on the trigger observations, where the output of the treatment and the control model are different. Although evaluation with full information about trigger observations (full knowledge) improves the precision, detecting all such trigger observations is a costly affair. In this paper, we propose a sampling based evaluation method (partial knowledge) to reduce this cost. The randomness of sampling introduces bias in the estimated outcome. We theoretically analyze this bias and show that the bias is inversely proportional to the number of observations used for sampling. We also compare the proposed evaluation methods using simulation and empirical data. In simulation, bias in evaluation with partial knowledge effectively reduces to zero when a limited number of observations (<= 0.1%) are sampled for trigger estimation. In empirical setup, evaluation with partial knowledge reduces the standard error by 36.48%.