Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth

TL;DR

This paper introduces a novel statistical framework that enhances the efficiency and robustness of A/B testing in business environments with small samples, non-Gaussian data, and ROI considerations.

Contribution

It proposes a doubly robust generalized U statistic that improves A/B testing by addressing small sample sizes, distributional issues, and confounding factors within a unified framework.

Findings

The new method outperforms standard t-tests in simulations.

Application to LinkedIn A/B tests demonstrates practical benefits.

Theoretical analysis confirms efficiency gains.

Abstract

The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we (i) show the statistical efficiency of using estimating equation and U statistics, which can address these issues separately; and (ii) propose a novel doubly robust generalized U that allows flexible definition of treatment effect, and can handles small samples, distribution robustness, ROI and confounding consideration in one framework. We provide theoretical results on asymptotics and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies, apply the methods to multiple real A/B tests at LinkedIn, and share results and learnings that are broadly useful.