Breaking the Winner's Curse with Bayesian Hybrid Shrinkage

TL;DR

This paper introduces a Bayesian hybrid shrinkage method to correct the Winner's Curse bias in large-scale online experiments, improving effect size estimation and confidence interval validity.

Contribution

It proposes a novel Bayesian approach with local shrinkage factors that enhances robustness and practicality for at-scale experimental analysis.

Findings

Performs well under various simulation scenarios.

Provides more accurate effect estimates.

Offers well-calibrated uncertainty quantification.

Abstract

A 'Winner's Curse' arises in large-scale online experimentation platforms when the same experiments are used to both select treatments and evaluate their effects. In these settings, classical difference-in-means estimators of treatment effects are upwardly biased and conventional confidence intervals are rendered invalid. The bias scales with the magnitude of sampling variability and the selection threshold, and inversely with the treatment's true effect size. We propose a new Bayesian approach that incorporates experiment-specific 'local shrinkage' factors that mitigate sensitivity to the choice of prior and improve robustness to assumption violations. We demonstrate how the associated posterior distribution can be estimated without numerical integration techniques, making it a practical choice for at-scale deployment. Through simulation, we evaluate the performance of our approach under various scenarios and find that it performs well even when assumptions about the sampling and selection processes are violated. In an empirical evaluation, our approach demonstrated superior performance over alternative methods, providing more accurate estimates with well-calibrated uncertainty quantification.