Bayes-Optimal Effort Allocation in Crowdsourcing: Bounds and Index Policies

TL;DR

This paper develops a computationally feasible approach to optimal effort allocation in crowdsourcing, providing bounds on the best possible accuracy and proposing an index policy that outperforms existing methods.

Contribution

It introduces a tractable upper bound on the Bayes-optimal policy and an index-based effort allocation policy inspired by restless bandits, advancing crowdsourcing efficiency.

Findings

The index policy outperforms existing methods in experiments.

The upper bound effectively estimates the optimal policy value.

The proposed approach is computationally feasible for practical use.

Abstract

We consider effort allocation in crowdsourcing, where we wish to assign labeling tasks to imperfect homogeneous crowd workers to maximize overall accuracy in a continuous-time Bayesian setting, subject to budget and time constraints. The Bayes-optimal policy for this problem is the solution to a partially observable Markov decision process, but the curse of dimensionality renders the computation infeasible. Based on the Lagrangian Relaxation technique in Adelman & Mersereau (2008), we provide a computationally tractable instance-specific upper bound on the value of this Bayes-optimal policy, which can in turn be used to bound the optimality gap of any other sub-optimal policy. In an approach similar in spirit to the Whittle index for restless multiarmed bandits, we provide an index policy for effort allocation in crowdsourcing and demonstrate numerically that it outperforms other stateof- arts and performs close to optimal solution.