The Significance Filter, the Winner's Curse and the Need to Shrink
Erik van Zwet, Eric Cator

TL;DR
This paper examines how the focus on statistically significant results, known as the significance filter, causes overestimation and undercoverage issues, especially with low power, and emphasizes the importance of shrinkage for accurate inference.
Contribution
It provides formal frequentist and Bayesian analyses of the significance filter's effects, including proofs of bias reduction with increased power and conditions for confidence interval undercoverage.
Findings
Bias decreases as power increases
Confidence intervals undercover when power is below 50%
Failure to shrink leads to misleading inferences
Abstract
The "significance filter" refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we condition on significance. In fact, the significance filter leads to overestimation of the magnitude of the parameter, which has been called the "winner's curse". It can also lead to undercoverage of the confidence interval. Moreover, these problems become more severe if the power is low. While these issues clearly deserve our attention, they have been studied only informally and mathematical results are lacking. Here we study them from the frequentist and the Bayesian perspective. We prove that the relative bias of the magnitude is a decreasing function of the power and that the usual confidence interval undercovers when the power is less than 50%. We…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
