Breaking the Winner's Curse with Bayesian Hybrid Shrinkage

TL;DR

This paper introduces Bayesian Hybrid Shrinkage, a novel empirical Bayes method that reduces the Winner's Curse bias in A/B testing by using data-driven, experiment-specific shrinkage, improving effect estimate accuracy and confidence interval validity.

Contribution

It presents a new Bayesian hybrid shrinkage approach with local, experiment-specific shrinkage factors and a closed-form inference method for high-throughput environments, enhancing bias correction.

Findings

BHS reduces bias more effectively than existing methods.

BHS maintains accurate confidence intervals under model violations.

Demonstrated improved performance on Meta's real-world data.

Abstract

The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit upwardly biased effect estimates and invalid confidence intervals. This selection bias leads to over-optimistic impact projections and undermines decision-making, particularly in low-power regimes. We propose Bayesian Hybrid Shrinkage (BHS), an empirical Bayes (EB) framework that leverages data-driven priors to mitigate selection bias and provides accurate uncertainty quantification. Unlike traditional EB methods that apply uniform shrinkage, BHS introduces an experiment-specific "local" shrinkage factor that incorporates individual experiment characteristics, improving robustness against prior misspecification. We also derive a closed-form inference strategy designed for high-throughput production environments. Extensive simulations and real-world evaluations at Meta Platforms demonstrate that BHS outperforms existing methods in terms of bias reduction and interval coverage, even under substantial violations of modeling assumptions.