Towards replicability with confidence intervals for the exceedance probability

TL;DR

This paper proposes confidence intervals for the exceedance probability to improve reproducibility and address issues with p-value misuse, offering a more informative uncertainty measure for replication studies.

Contribution

It introduces a novel method for constructing confidence intervals for exceedance probabilities, enhancing the tools available for replication and scientific inference.

Findings

Confidence intervals provide clearer uncertainty than p-values.

The method is applicable to linear models and relates to existing p-value and confidence interval frameworks.

Examples demonstrate the approach with simulated and real data.

Abstract

Several scientific fields including psychology are undergoing a replication crisis. There are many reasons for this problem, one of which is a misuse of p-values. There are several alternatives to p-values, and in this paper we describe a complement that is geared towards replication. In particular, we focus on confidence intervals for the probability that a parameter estimate will exceed a specified value in an exact replication study. These intervals convey uncertainty in a way that p-values and standard confidence intervals do not, and can help researchers to draw sounder scientific conclusions. After briefly reviewing background on p-values and a few alternatives, we describe our approach and provide examples with simulated and real data. For linear models, we also describe how confidence intervals for the exceedance probability are related to p-values and confidence intervals for parameters.