On Post-Selection Inference in A/B Tests

TL;DR

This paper introduces novel post-selection inference methods for A/B testing that combine machine learning and empirical Bayes techniques to improve the reliability of statistical conclusions.

Contribution

It proposes new combined approaches for post-selection inference in A/B tests, enhancing bias reduction and confidence interval accuracy.

Findings

Reduced post-selection biases in A/B testing

Improved confidence interval coverage rates

Effective in large-scale simulated and real data

Abstract

When interpreting A/B tests, we typically focus only on the statistically significant results and take them by face value. This practice, termed post-selection inference in the statistical literature, may negatively affect both point estimation and uncertainty quantification, and therefore hinder trustworthy decision making in A/B testing. To address this issue, in this paper we explore two seemingly unrelated paths, one based on supervised machine learning and the other on empirical Bayes, and propose post-selection inferential approaches that combine the strengths of both. Through large-scale simulated and empirical examples, we demonstrate that our proposed methodologies stand out among other existing ones in both reducing post-selection biases and improving confidence interval coverage rates, and discuss how they can be conveniently adjusted to real-life scenarios.