Consider avoiding the .05 significance level

TL;DR

The paper argues that the conventional p value threshold of .05 in NHST should be avoided, proposing smaller thresholds for more reliable evidence, based on Bayesian posterior probabilities.

Contribution

It provides a Bayesian perspective on NHST, illustrating how smaller p value thresholds yield more consistent posterior probabilities, challenging the standard .05 cutoff.

Findings

Posterior probability remains high for p = .05 with small N.

Lower p values like .001 significantly reduce posterior probability of H0.

Very small p values like .0001 make posterior probability nearly independent of N.

Abstract

It is suggested that some shortcomings of Null Hypothesis Significance Testing (NHST), viewed from the perspective of Bayesian statistics, turn benign once the traditional threshold p value of .05 is substituted by a sufficiently smaller value. To illustrate, the posterior probability of H0 stating P=.5, given data that just render it rejected by NHST with a p value of .05 (and a uniform prior), is shown here to be not much smaller than .50 for most values of N below 100 (and even exceeds .50 for N>=100); in contrast, with a p value of .001 posterior probability does not exceed .06 for N<=100 (neither .25 for N<9000). Yet more interesting, posterior probability becomes quite independent of N with a p value of .0001, hence practically satisfying the alpha postulate - set by Cornfield (1966) as the condition for p value being a measure of evidence in itself. In view of the low prospect that most researchers will soon convert to use Bayesian statistics in any form, we thus suggest that researchers who elect the conservative option of resorting to NHST be encouraged to avoid as much as possible using a p value of .05 as a threshold for rejecting H0. The analysis presented here may be used to discuss afresh which level of threshold p value seems to be a reasonable, practical substitute.