Statistical Assessment of Replicability via Bayesian Model Criticism

TL;DR

This paper introduces Bayesian model criticism methods to evaluate the replicability of scientific results, aiming to improve research reliability through statistical assessment and real data examples.

Contribution

It proposes two Bayesian model criticism approaches for assessing irreproducibility, extending existing methods and applicable to diverse scientific scenarios.

Findings

Methods effectively identify irreproducible results

Applications include psychology and COVID-19 data analyses

Statistical properties support robustness of approaches

Abstract

Assessment of replicability is critical to ensure the quality and rigor of scientific research. In this paper, we discuss inference and modeling principles for replicability assessment. Targeting distinct application scenarios, we propose two types of Bayesian model criticism approaches to identify potentially irreproducible results in scientific experiments. They are motivated by established Bayesian prior and posterior predictive model-checking procedures and generalize many existing replicability assessment methods. Finally, we discuss the statistical properties of the proposed replicability assessment approaches and illustrate their usages by simulations and examples of real data analysis, including the data from the Reproducibility Project: Psychology and a systematic review of impacts of pre-existing cardiovascular disease on COVID-19 outcomes.