Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks
Veniamin Veselovsky, Manoel Horta Ribeiro, Robert West

TL;DR
This study reveals that a significant portion of crowd workers on Amazon Mechanical Turk use large language models to complete text summarization tasks, raising concerns about data authenticity and the impact on research quality.
Contribution
The paper introduces a methodology combining keystroke detection and synthetic text classification to estimate LLM usage among crowd workers, highlighting the need for new safeguards.
Findings
33-46% of crowd workers used LLMs for task completion
LLM usage impacts the validity of human-generated annotations
Proposes a method to detect LLM usage in crowdsourcing tasks
Abstract
Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · AI in Service Interactions · Text Readability and Simplification
