Replicability: Terminology, Measuring Success, and Strategy

TL;DR

This paper discusses the importance of clear terminology, relevance thresholds, and strategies for successful scientific replication, emphasizing effect size estimation and heterogeneity in empirical research.

Contribution

It clarifies key issues in replication, proposing a focus on effect size relevance and heterogeneity, and offers a strategy to improve replicability in empirical science.

Findings

Effect size estimation with precision is crucial for scientific claims.

Testing for effect relevance is more meaningful than zero-effect tests.

Multiple replications are needed to assess heterogeneity accurately.

Abstract

Empirical science needs to be based on facts and claims that can be reproduced. This calls for replicating the studies that proclaim the claims, but practice in most fields still fails to implement this idea. When such studies emerged in the past decade, the results were generally disappointing. There have been an overwhelming number of papers addressing the ``reproducibility crisis'' in the last 20 years. Nevertheless, terminology is not yet settled, and there is no consensus about when a replication should be called successful. This paper intends to clarify such issues. A fundamental problem in empirical science is that usual claims only state that effects are non-zero, and such statements are scientifically void. An effect must have a \emph{relevant} size to become a reasonable item of knowledge. Therefore, estimation of an effect, with an indication of precision, forms a substantial scientific task, whereas testing it against zero does not. A relevant effect is one that is shown to exceed a relevance threshold. This paradigm has implications for the judgement on replication success. A further issue is the unavoidable variability between studies, called heterogeneity in meta-analysis. Therefore, it is of little value, again, to test for zero difference between an original effect and its replication, but exceedance of a corresponding relevance threshold should be tested. In order to estimate the degree of heterogeneity, more than one replication is needed, and an appropriate indication of the precision of an estimated effect requires such an estimate. These insights, which are discussed in the paper, show the complexity of obtaining solid scientific results, implying the need for a strategy to make replication happen.