On the state of reporting in crowdsourcing experiments and a checklist to aid current practices

TL;DR

This paper analyzes how crowdsourcing experiments are reported, highlighting gaps and proposing a checklist to improve transparency, reproducibility, and fair assessment of research outcomes in human subject studies.

Contribution

It provides an analysis of current reporting practices and introduces a comprehensive checklist to enhance reporting standards in crowdsourcing experiments.

Findings

Many crowdsourcing experiments are under-reported.

The proposed checklist aims to standardize reporting practices.

Improved reporting can enhance reproducibility and fairness.

Abstract

Crowdsourcing is being increasingly adopted as a platform to run studies with human subjects. Running a crowdsourcing experiment involves several choices and strategies to successfully port an experimental design into an otherwise uncontrolled research environment, e.g., sampling crowd workers, mapping experimental conditions to micro-tasks, or ensure quality contributions. While several guidelines inform researchers in these choices, guidance of how and what to report from crowdsourcing experiments has been largely overlooked. If under-reported, implementation choices constitute variability sources that can affect the experiment's reproducibility and prevent a fair assessment of research outcomes. In this paper, we examine the current state of reporting of crowdsourcing experiments and offer guidance to address associated reporting issues. We start by identifying sensible implementation choices, relying on existing literature and interviews with experts, to then extensively analyze the reporting of 171 crowdsourcing experiments. Informed by this process, we propose a checklist for reporting crowdsourcing experiments.