The Assessment of Intrinsic Credibility and a New Argument for p<0.005

TL;DR

This paper evaluates the concept of intrinsic credibility of statistical findings, proposing Bayesian and frequentist methods to assess whether results are trustworthy and introducing a new p-value threshold close to p<0.005.

Contribution

It introduces Bayesian prior-predictive tail probabilities and the credibility ratio as novel tools for assessing intrinsic credibility of findings.

Findings

Proposes a new p-value threshold near 0.005 for intrinsic credibility.

Introduces the credibility ratio, requiring it to be less than 5.8 for credible results.

Defines a p-value for intrinsic credibility with a clear frequentist interpretation.

Abstract

The concept of intrinsic credibility has been recently introduced to check the credibility of "out of the blue" findings without any prior support. A significant result is deemed intrinsically credible if it is in conflict with a sceptical prior derived from the very same data that would make the effect non-significant. In this paper I propose to use Bayesian prior-predictive tail probabilities to assess intrinsic credibility. For the standard 5% significance level, this leads to a new p-value threshold that is remarkably close to the recently proposed p<0.005 standard. I also introduce the credibility ratio, the ratio of the upper to the lower limit of a standard confidence interval for the corresponding effect size. I show that the credibility ratio has to be smaller than 5.8 such that a significant finding is also intrinsically credible. Finally, a p-value for intrinsic credibility is proposed that is a simple function of the ordinary p-value and has a direct frequentist interpretation in terms of the probability of replicating an effect.