Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

TL;DR

This study investigates whether laypeople can predict the replicability of social science studies without expert help, finding they are engaged and understand the material but have limited predictive accuracy.

Contribution

It demonstrates that laypeople can engage with raw research materials and provide reasonable responses, but their predictions are less accurate than those involving expert interpretation.

Findings

Laypeople showed good engagement and understanding.

Their predictions had limited accuracy for actual replication outcomes.

Expert interpretation improves predictive accuracy.

Abstract

The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.