Working in Pairs: Understanding the Effects of Worker Interactions in Crowdwork

TL;DR

This paper investigates how peer communication between workers in crowdsourcing tasks influences work quality, finding significant improvements in task accuracy but limited impact on future independent performance.

Contribution

It introduces the concept of peer communication in crowdsourcing, demonstrating its effects on various task types and providing experimental evidence of its benefits.

Findings

Peer communication significantly improves work quality.

Limited effect on future independent performance.

Effective across multiple task types.

Abstract

Crowdsourcing has gained popularity as a tool to harness human brain power to help solve problems that are difficult for computers. Previous work in crowdsourcing often assumes that workers complete crowdwork independently. In this paper, we relax the independent property of crowdwork and explore how introducing direct, synchronous, and free-style interactions between workers would affect crowdwork. In particular, motivated by the concept of peer instruction in educational settings, we study the effects of peer communication in crowdsourcing environments. In the crowdsourcing setting with peer communication, pairs of workers are asked to complete the same task together by first generating their initial answers to the task independently and then freely discussing the tasks with each other and updating their answers after the discussion. We experimentally examine the effects of peer communication in crowdwork on various common types of tasks on crowdsourcing platforms, including image labeling, optical character recognition (OCR), audio transcription, and nutrition analysis. Our experiment results show that the work quality is significantly improved in tasks with peer communication compared to tasks where workers complete the work independently. However, participating in tasks with peer communication has limited effects on influencing worker's independent performance in tasks of the same type in the future.