Large Scale Replication Projects in Contemporary Psychological Research
Blakeley B. McShane, Jennifer L. Tackett, Ulf Bockenholt and, Andrew Gelman

TL;DR
This paper discusses the limitations of current large-scale psychological replication projects, emphasizing the importance of embracing heterogeneity to better assess true replicability and improve underlying theories.
Contribution
It advocates for incorporating heterogeneity into replication studies to more accurately evaluate effect variability and enhance psychological theories.
Findings
Heterogeneity in effect sizes is often underestimated in current projects.
Embracing heterogeneity can reveal issues with the replicability of psychological phenomena.
Revising theories in light of heterogeneity can improve scientific understanding.
Abstract
Replication is complicated in psychological research because studies of a given psychological phenomenon can never be direct or exact replications of one another, and thus effect sizes vary from one study of the phenomenon to the next--an issue of clear importance for replication. Current large scale replication projects represent an important step forward for assessing replicability, but provide only limited information because they have thus far been designed in a manner such that heterogeneity either cannot be assessed or is intended to be eliminated. Consequently, the non-trivial degree of heterogeneity found in these projects represents a lower bound on heterogeneity. We recommend enriching large scale replication projects going forward by em- bracing heterogeneity. We argue this is key for assessing replicability: if effect sizes are sufficiently heterogeneous--even if the sign of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
