Learning to Hire Teams

TL;DR

This paper develops online learning algorithms to efficiently identify and hire effective teams for crowdsourcing tasks, leveraging task and worker similarities to minimize costs and improve hiring accuracy.

Contribution

It introduces decision-theoretic algorithms with PAC bounds for optimal team hiring, incorporating graph-based side observations to accelerate learning.

Findings

Algorithms achieve near-optimal team selection with high confidence.

Graph structures improve learning efficiency and reduce costs.

Validated on simulated and real-world crowdsourcing data.

Abstract

Crowdsourcing and human computation has been employed in increasingly sophisticated projects that require the solution of a heterogeneous set of tasks. We explore the challenge of building or hiring an effective team, for performing tasks required for such projects on an ongoing basis, from an available pool of applicants or workers who have bid for the tasks. The recruiter needs to learn workers' skills and expertise by performing online tests and interviews, and would like to minimize the amount of budget or time spent in this process before committing to hiring the team. How can one optimally spend budget to learn the expertise of workers as part of recruiting a team? How can one exploit the similarities among tasks as well as underlying social ties or commonalities among the workers for faster learning? We tackle these decision-theoretic challenges by casting them as an instance of online learning for best action selection. We present algorithms with PAC bounds on the required budget to hire a near-optimal team with high confidence. Furthermore, we consider an embedding of the tasks and workers in an underlying graph that may arise from task similarities or social ties, and that can provide additional side-observations for faster learning. We then quantify the improvement in the bounds that we can achieve depending on the characteristic properties of this graph structure. We evaluate our methodology on simulated problem instances as well as on real-world crowdsourcing data collected from the oDesk platform. Our methodology and results present an interesting direction of research to tackle the challenges faced by a recruiter for contract-based crowdsourcing.