The Unreasonable Effectiveness of Monte Carlo Simulations in A/B Testing

TL;DR

This paper demonstrates how Monte Carlo simulations are a powerful and versatile tool for understanding, validating, and improving A/B testing and RCT methodologies, including false positive rates, power estimation, and network effects.

Contribution

It provides a comprehensive analysis of Monte Carlo simulation applications in A/B testing, including variance reduction, early stopping, and comparisons of statistical approaches.

Findings

Simulations help understand false positive rates and statistical power.

Monte Carlo methods clarify relationships between p-values and posterior probabilities.

Simulations can model network effects in social network RCTs.

Abstract

This paper examines the use of Monte Carlo simulations to understand statistical concepts in A/B testing and Randomized Controlled Trials (RCTs). We discuss the applicability of simulations in understanding false positive rates and estimate statistical power, implementing variance reduction techniques and examining the effects of early stopping. By comparing frequentist and Bayesian approaches, we illustrate how simulations can clarify the relationship between p-values and posterior probabilities, and the validity of such approximations. The study also references how Monte Carlo simulations can be used to understand network effects in RCTs on social networks. Our findings show that Monte Carlo simulations are an effective tool for experimenters to deepen their understanding and ensure their results are statistically valid and practically meaningful.