Community-driven reviewing and validation of publications

TL;DR

This paper discusses the authors' long-term practical experience with crowdsourcing for research evaluation and reviewing, highlighting challenges like reproducibility and potential solutions.

Contribution

It provides insights into community-driven review processes and shares practical lessons learned over a decade.

Findings

Crowdsourcing can effectively supplement traditional peer review.

Reproducibility remains a significant challenge in research validation.

Community involvement enhances review diversity and coverage.

Abstract

In this report, we share our practical experience on crowdsourcing evaluation of research artifacts and reviewing of publications since 2008. We also briefly discuss encountered problems including reproducibility of experimental results and possible solutions.