The Significance Filter, the Winner's Curse and the Need to Shrink

TL;DR

This paper examines how the focus on statistically significant results, known as the significance filter, causes overestimation and undercoverage issues, especially with low power, and emphasizes the importance of shrinkage for accurate inference.

Contribution

It provides formal frequentist and Bayesian analyses of the significance filter's effects, including proofs of bias reduction with increased power and conditions for confidence interval undercoverage.

Findings

Bias decreases as power increases

Confidence intervals undercover when power is below 50%

Failure to shrink leads to misleading inferences

Abstract

The "significance filter" refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we condition on significance. In fact, the significance filter leads to overestimation of the magnitude of the parameter, which has been called the "winner's curse". It can also lead to undercoverage of the confidence interval. Moreover, these problems become more severe if the power is low. While these issues clearly deserve our attention, they have been studied only informally and mathematical results are lacking. Here we study them from the frequentist and the Bayesian perspective. We prove that the relative bias of the magnitude is a decreasing function of the power and that the usual confidence interval undercovers when the power is less than 50%. We conclude that failure to apply the appropriate amount of shrinkage can lead to misleading inferences.