Strategic A/B testing via Maximum Probability-driven Two-armed Bandit

TL;DR

This paper introduces a novel two-armed bandit approach for A/B testing that improves detection of minor effects by leveraging a maximum probability-driven framework, permutation methods, and a strategic CLT, leading to higher power and cost efficiency.

Contribution

It proposes a new maximum probability-driven two-armed bandit method with permutation-based robustness, enhancing sensitivity to small effects in A/B testing.

Findings

Improved detection of minor effects in A/B testing.

Enhanced statistical power and reduced costs.

More concentrated null distribution under the new method.

Abstract

Detecting a minor average treatment effect is a major challenge in large-scale applications, where even minimal improvements can have a significant economic impact. Traditional methods, reliant on normal distribution-based or expanded statistics, often fail to identify such minor effects because of their inability to handle small discrepancies with sufficient sensitivity. This work leverages a counterfactual outcome framework and proposes a maximum probability-driven two-armed bandit (TAB) process by weighting the mean volatility statistic, which controls Type I error. The implementation of permutation methods further enhances the robustness and efficacy. The established strategic central limit theorem (SCLT) demonstrates that our approach yields a more concentrated distribution under the null hypothesis and a less concentrated one under the alternative hypothesis, greatly improving statistical power. The experimental results indicate a significant improvement in the A/B testing, highlighting the potential to reduce experimental costs while maintaining high statistical power.