Tuning the Diversity of Open-Ended Responses from the Crowd

TL;DR

This paper introduces a mechanism for crowdsourcing that incentivizes workers to balance exploring new solutions and evaluating existing ones, improving the reliability and efficiency of open-ended response collection.

Contribution

It proposes a novel game-theoretic mechanism with distinct payoffs for proposing, voting, or abstaining, enabling better control over response diversity and quality in crowdsourcing.

Findings

Mechanism effectively balances answer discovery and convergence.

Experimental results demonstrate reliable control over response diversity.

System improves crowd response quality and efficiency.

Abstract

Crowdsourcing can solve problems that current fully automated systems cannot. Its effectiveness depends on the reliability, accuracy, and speed of the crowd workers that drive it. These objectives are frequently at odds with one another. For instance, how much time should workers be given to discover and propose new solutions versus deliberate over those currently proposed? How do we determine if discovering a new answer is appropriate at all? And how do we manage workers who lack the expertise or attention needed to provide useful input to a given task? We present a mechanism that uses distinct payoffs for three possible worker actions---propose,vote, or abstain---to provide workers with the necessary incentives to guarantee an effective (or even optimal) balance between searching for new answers, assessing those currently available, and, when they have insufficient expertise or insight for the task at hand, abstaining. We provide a novel game theoretic analysis for this mechanism and test it experimentally on an image---labeling problem and show that it allows a system to reliably control the balance betweendiscovering new answers and converging to existing ones.