Power-up! What Can Generative Models Do for Human Computation Workflows?

TL;DR

This paper explores how large language models can enhance human computation workflows, identifying key integration points and proposing design improvements to optimize hybrid human-AI crowdsourcing systems.

Contribution

It introduces a framework for integrating LLMs into crowdsourcing workflows and discusses potential benefits and evaluation strategies for such hybrid systems.

Findings

Identifies workflow junctures for LLM integration

Proposes design patterns for hybrid crowd work

Highlights research gaps in LLM-assisted crowdsourcing

Abstract

We are amidst an explosion of artificial intelligence research, particularly around large language models (LLMs). These models have a range of applications across domains like medicine, finance, commonsense knowledge graphs, and crowdsourcing. Investigation into LLMs as part of crowdsourcing workflows remains an under-explored space. The crowdsourcing research community has produced a body of work investigating workflows and methods for managing complex tasks using hybrid human-AI methods. Within crowdsourcing, the role of LLMs can be envisioned as akin to a cog in a larger wheel of workflows. From an empirical standpoint, little is currently understood about how LLMs can improve the effectiveness of crowdsourcing workflows and how such workflows can be evaluated. In this work, we present a vision for exploring this gap from the perspectives of various stakeholders involved in the crowdsourcing paradigm -- the task requesters, crowd workers, platforms, and end-users. We identify junctures in typical crowdsourcing workflows at which the introduction of LLMs can play a beneficial role and propose means to augment existing design patterns for crowd work.