The world of research has gone berserk: modeling the consequences of requiring "greater statistical stringency" for scientific publication

TL;DR

This paper models how implementing stricter statistical standards in scientific publishing could influence research practices and the reliability of published results, providing insights into the potential benefits and drawbacks of such measures.

Contribution

It introduces a novel optimality model to predict researchers' strategies under increased statistical stringency and offers a methodology to estimate the reliability of published research.

Findings

Model predicts changes in research strategies with increased statistical stringency.

Method estimates the reliability of published research considering diverse strategies.

Analysis informs the debate on adopting stricter statistical standards.

Abstract

In response to growing concern about the reliability and reproducibility of published science, researchers have proposed adopting measures of greater statistical stringency, including suggestions to require larger sample sizes and to lower the highly criticized p<0.05 significance threshold. While pros and cons are vigorously debated, there has been little to no modeling of how adopting these measures might affect what type of science is published. In this paper, we develop a novel optimality model that, given current incentives to publish, predicts a researcher's most rational use of resources in terms of the number of studies to undertake, the statistical power to devote to each study, and the desirable pre-study odds to pursue. We then develop a methodology that allows one to estimate the reliability of published research by considering a distribution of preferred research strategies. Using this approach, we investigate the merits of adopting measures of `greater statistical stringency' with the goal of informing the ongoing debate.