Do t-Statistic Hurdles Need to be Raised?

TL;DR

This paper critically examines the justification for raising statistical significance thresholds, showing that biases in published data complicate such efforts, while methods targeting published results are more reliably identified.

Contribution

It provides a theoretical and empirical analysis demonstrating the challenges in justifying higher statistical hurdles due to bias and weak identification, and highlights the reliability of methods focusing on published findings.

Findings

Bias in published data complicates raising significance thresholds.

Methods targeting published results are more strongly identified.

Empirical analysis on return predictability supports theoretical claims.

Abstract

Many scholars have called for raising statistical hurdles to guard against false discoveries in academic publications. I show these calls may be difficult to justify empirically. Published data exhibit bias: results that fail to meet existing hurdles are often unobserved. These unobserved results must be extrapolated, which can lead to weak identification of revised hurdles. In contrast, statistics that can target only published findings (e.g. empirical Bayes shrinkage and the FDR) can be strongly identified, as data on published findings is plentiful. I demonstrate these results theoretically and in an empirical analysis of the cross-sectional return predictability literature.