Can we trust online crowdworkers? Comparing online and offline participants in a preference test of virtual agents

TL;DR

This study compares in-lab and online crowdworker evaluations of virtual agent gestures, finding online platforms produce reliable perceptual data comparable to traditional lab experiments, thus validating their use in such research.

Contribution

It provides empirical evidence that online crowdsourcing platforms can reliably replace in-lab experiments for perceptual evaluations of virtual agents.

Findings

No significant difference in evaluations across participant pools

Online platforms yield reliable perceptual data

Crowdsourcing is a viable alternative for user studies

Abstract

Conducting user studies is a crucial component in many scientific fields. While some studies require participants to be physically present, other studies can be conducted both physically (e.g. in-lab) and online (e.g. via crowdsourcing). Inviting participants to the lab can be a time-consuming and logistically difficult endeavor, not to mention that sometimes research groups might not be able to run in-lab experiments, because of, for example, a pandemic. Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) or Prolific can therefore be a suitable alternative to run certain experiments, such as evaluating virtual agents. Although previous studies investigated the use of crowdsourcing platforms for running experiments, there is still uncertainty as to whether the results are reliable for perceptual studies. Here we replicate a previous experiment where participants evaluated a gesture generation model for virtual agents. The experiment is conducted across three participant pools -- in-lab, Prolific, and AMT -- having similar demographics across the in-lab participants and the Prolific platform. Our results show no difference between the three participant pools in regards to their evaluations of the gesture generation models and their reliability scores. The results indicate that online platforms can successfully be used for perceptual evaluations of this kind.