Replication of "null results" -- Absence of evidence or evidence of absence?

TL;DR

This paper critiques the common practice of interpreting non-significant results in replication studies as success, emphasizing the need for proper methods like equivalence testing and Bayes factors to accurately assess evidence for the absence of effects.

Contribution

It highlights the logical flaws in equating non-significance with replication success and proposes rigorous statistical methods to properly evaluate evidence for null effects in replication research.

Findings

Many null results are inconclusive rather than evidence of no effect

Equivalence testing and Bayes factors provide better evidence assessment

Proper design and analysis are crucial for valid null result interpretation

Abstract

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a "replication success". Here we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and "replication success" can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with "null results" are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.