Valuing Winners: When and How to Correct for Selection Bias in Randomized Experiments

TL;DR

This paper analyzes the winner's curse in randomized experiments, distinguishing two bias types, evaluating correction methods, and proposing an adaptive confidence interval procedure tailored to decision-maker objectives.

Contribution

It introduces a comprehensive framework for understanding and correcting winner's curse bias, and proposes an adaptive empirical likelihood method for valid confidence intervals.

Findings

No single correction method dominates across all scenarios.

Plug-in estimator excels with large treatment differences.

Cross-fitting performs best with similar treatments.

Abstract

Decision-makers often deploy the best-performing treatment from a randomized experiment, creating a winner's curse: selection favors treatments whose observed outcomes are high partly because of statistical noise, so the na\"ive estimate of the winner is upward biased. We distinguish two forms of winner's curse, bias relative to the true best treatment (global) and bias relative to the selected treatment's true mean (selective), and link them to regret from deploying a suboptimal treatment. This framework defines seven decision-relevant evaluation targets: mean bias, mean squared error, and confidence interval coverage for the global and selective winner's curse, and mean regret. We then show that methods that perform well on one target can perform poorly on others, so corrections should be matched to the manager's objective. Across simulations with varying effect sizes, multiple-arm settings, and data calibrated to an online A/B testing platform, no method dominates uniformly: the plug-in estimator performs best when treatment differences are large, cross-fitting performs best when treatments are similar, and resampling methods often achieve low mean squared error for moderate differences. We also introduce an adaptive empirical likelihood procedure that delivers asymptotically valid confidence intervals across settings without the tuning sensitivity of resampling-based methods.