p-Values for Credibility

TL;DR

This paper formalizes a reverse-Bayes approach to assess the credibility of significance results using Bayesian predictive tail probabilities, introducing p-values for extrinsic and intrinsic credibility that relate to effect size and replication probability.

Contribution

It introduces a formal Bayesian framework for credibility assessment, including new p-values for extrinsic and intrinsic credibility, and links significance thresholds to replication probability.

Findings

Proposes a p-value for extrinsic credibility combining internal and external evidence.

Introduces a p-value for intrinsic credibility based on the ordinary p-value.

Suggests a significance threshold close to 0.005 for intrinsic credibility.

Abstract

Analysis of credibility is a reverse-Bayes technique that has been proposed by Matthews (2001) to overcome some of the shortcomings of significance tests. A significant result is deemed credible if current knowledge about the effect size is in conflict with any sceptical prior that would make the effect non-significant. In this paper I formalize the approach and propose to use Bayesian predictive tail probabilities to quantify the evidence for credibility. This gives rise to a p-value for extrinsic credibility, taking into account both the internal and the external evidence for an effect. The assessment of intrinsic credibility leads to a new threshold for ordinary significance that is remarkably close to the recently proposed 0.005 level. Finally, a p-value for intrinsic credibility is proposed that is a simple function of the ordinary p-value for significance and has a direct frequentist interpretation in terms of the replication probability that a future study under identical conditions will give an estimated effect in the same direction as the first study.