Learning to Incentivize: Eliciting Effort via Output Agreement

TL;DR

This paper studies how to set optimal rewards in output agreement mechanisms to incentivize effort and truthful answers in crowdsourcing, considering workers with heterogeneous effort costs and unknown cost distributions.

Contribution

It derives the optimal reward level when the cost distribution is known and develops sequential mechanisms to learn and optimize rewards when it is unknown.

Findings

Optimal reward levels are characterized for known cost distributions.

Sequential mechanisms effectively learn and approximate the optimal reward in unknown settings.

The approach improves effort elicitation in crowdsourcing without direct answer verification.

Abstract

In crowdsourcing when there is a lack of verification for contributed answers, output agreement mechanisms are often used to incentivize participants to provide truthful answers when the correct answer is hold by the majority. In this paper, we focus on using output agreement mechanisms to elicit effort, in addition to eliciting truthful answers, from a population of workers. We consider a setting where workers have heterogeneous cost of effort exertion and examine the data requester's problem of deciding the reward level in output agreement for optimal elicitation. In particular, when the requester knows the cost distribution, we derive the optimal reward level for output agreement mechanisms. This is achieved by first characterizing Bayesian Nash equilibria of output agreement mechanisms for a given reward level. When the requester does not know the cost distribution, we develop sequential mechanisms that combine learning the cost distribution with incentivizing effort exertion to approximately determine the optimal reward level.