Why multiple hypothesis test corrections provide poor control of false positives in the real world

TL;DR

This paper critiques the effectiveness of multiple hypothesis test corrections in real-world scientific research, highlighting their poor control over false positives due to flexible analysis practices and proposing alternative approaches like shrinking parameter estimates.

Contribution

The paper provides a critical analysis of the limitations of traditional multiple testing corrections and advocates for alternative methods such as shrinking estimates for better false positive control.

Findings

False positive rates are often much higher than nominal levels.

Traditional p-value adjustments are unreliable in practice.

Shrinking parameter estimates can improve inference accuracy.

Abstract

Most scientific disciplines use significance testing to draw conclusions about experimental or observational data. This classical approach provides a theoretical guarantee for controlling the number of false positives across a set of hypothesis tests, making it an appealing framework for scientists seeking to limit the number of false effects or associations that they claim to observe. Unfortunately, this theoretical guarantee applies to few experiments, and the true false positive rate (FPR) is much higher. Scientists have plenty of freedom to choose the error rate to control, the tests to include in the adjustment, and the method of correction, making strong error control difficult to attain. In addition, hypotheses are often tested after finding unexpected relationships or patterns, the data are analysed in several ways, and analyses may be run repeatedly as data accumulate. As a result, adjusted p-values are too small, incorrect conclusions are often reached, and results are harder to reproduce. In the following, I argue why the FPR is rarely controlled meaningfully and why shrinking parameter estimates is preferable to p-value adjustments.