How to Tell When a Result Will Replicate: Significance and Replication in Distributional Null Hypothesis Tests

TL;DR

This paper introduces a distributional null hypothesis testing method that accounts for both within- and between-experiment variation, improving the prediction of result replicability over standard significance tests.

Contribution

It extends traditional significance testing by analyzing between-experiment variation, providing a more accurate estimate of replication probability.

Findings

Many significant results are consistent with random variation when between-experiment variation is considered.

Predicted replication probabilities strongly correlate with actual replication rates.

The new method offers a more reliable statistical tool for assessing result replicability.

Abstract

There is a well-known problem in Null Hypothesis Significance Testing: many statistically significant results fail to replicate in subsequent experiments. We show that this problem arises because standard `point-form null' significance tests consider only within-experiment but ignore between-experiment variation, and so systematically underestimate the degree of random variation in results. We give an extension to standard significance testing that addresses this problem by analysing both within- and between-experiment variation. This `distributional null' approach does not underestimate experimental variability and so is not overconfident in identifying significance; because this approach addresses between-experiment variation, it gives mathematically coherent estimates for the probability of replication of significant results. Using a large-scale replication dataset (the first `Many Labs' project), we show that many experimental results that appear statistically significant in standard tests are in fact consistent with random variation when both within- and between-experiment variation are taken into account in this approach. Further, grouping experiments in this dataset into `predictor-target' pairs we show that the predicted replication probabilities for target experiments produced in this approach (given predictor experiment results and the sample sizes of the two experiments) are strongly correlated with observed replication rates. Distributional null hypothesis testing thus gives researchers a statistical tool for identifying statistically significant and reliably replicable results.