Crowdsourcing Paper Screening in Systematic Literature Reviews
Evgeny Krivosheev, Fabio Casati, Valentina Caforio, Boualem Benatallah

TL;DR
This paper investigates the use of crowdsourcing to streamline the screening phase of systematic literature reviews, aiming to reduce effort and identify effective strategies within budget constraints.
Contribution
It introduces methods for optimizing crowdsourcing strategies in literature review screening, including test sizing, labeling, classification, and phased approaches, evaluated through experiments.
Findings
Crowdsourcing can effectively assist in literature review screening.
Phased crowdsourcing improves result quality and cost-efficiency.
Optimal test sizes and labeling strategies depend on budget constraints.
Abstract
Literature reviews allow scientists to stand on the shoulders of giants, showing promising directions, summarizing progress, and pointing out existing challenges in research. At the same time conducting a systematic literature review is a laborious and consequently expensive process. In the last decade, there have a few studies on crowdsourcing in literature reviews. This paper explores the feasibility of crowdsourcing for facilitating the literature review process in terms of results, time and effort, as well as to identify which crowdsourcing strategies provide the best results based on the budget available. In particular we focus on the screening phase of the literature review process and we contribute and assess methods for identifying the size of tests, labels required per paper, and classification functions as well as methods to split the crowdsourcing process in phases to improve…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
