A Flexible Defense Against the Winner's Curse

TL;DR

This paper introduces the zoom correction, a flexible statistical method for valid inference on the top-performing candidate, addressing the winner's curse across various fields and settings.

Contribution

The authors propose a novel, adaptable approach called zoom correction that corrects for selection bias in winner inference in both parametric and nonparametric contexts.

Findings

Effective correction for winner's curse demonstrated

Applicable to top k winners and near-winners

Handles arbitrary dependencies between candidates

Abstract

Across science and policy, decision-makers often need to draw conclusions about the best candidate among competing alternatives. For instance, researchers may seek to infer the effectiveness of the most successful treatment or determine which demographic group benefits most from a specific treatment. Similarly, in machine learning, practitioners are often interested in the population performance of the model that performs best empirically. However, cherry-picking the best candidate leads to the winner's curse: the observed performance for the winner is biased upwards, rendering conclusions based on standard measures of uncertainty invalid. We introduce the zoom correction, a novel approach for valid inference on the winner. Our method is flexible: it can be employed in both parametric and nonparametric settings, can handle arbitrary dependencies between candidates, and automatically adapts to the level of selection bias. The method easily extends to important related problems, such as inference on the top k winners, inference on the value and identity of the population winner, and inference on "near-winners."