Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests

TL;DR

This paper introduces a decision-theoretic, cost-aware ranking method for large-scale A/B testing that maximizes profit while controlling false discovery costs, demonstrating improved performance over existing methods.

Contribution

It develops a novel empirical Bayes framework with a greedy knapsack algorithm for optimal experiment ranking based on lift-to-cost ratio and false discovery control.

Findings

The proposed method is rank-optimal and valid.

It outperforms existing FDR-controlling methods in finite samples.

Application to Optimizely data shows significant business value.

Abstract

A/B testing is a core tool for decision-making in business experimentation, particularly in digital platforms and marketplaces. Practitioners often prioritize lift in performance metrics while seeking to control the costs of false discoveries. This paper develops a decision-theoretic framework for maximizing expected profit subject to a constraint on the cost-weighted false discovery rate (FDR). We propose an empirical Bayes approach that uses a greedy knapsack algorithm to rank experiments based on the ratio of expected lift to cost, incorporating the local false discovery rate (lfdr) as a key statistic. The resulting oracle rule is valid and rank-optimal. In large-scale settings, we establish the asymptotic validity of a data-driven implementation and demonstrate superior finite-sample performance over existing FDR-controlling methods. An application to A/B tests run on the Optimizely platform highlights the business value of the approach.