Incentivizing an Unknown Crowd

TL;DR

This paper introduces a reinforcement learning approach for sequentially eliciting information from an unknown, heterogeneous crowd in crowdsourcing, effectively handling irrationality and collusion without verification.

Contribution

It presents the first RL-based method for EIWV that adapts dynamically, uses a costly oracle, and is robust against collusion and irrational behaviors.

Findings

Effective against diverse worker behaviors

Robust to collusion and irrationality

Validated on large-scale real datasets

Abstract

Motivated by the common strategic activities in crowdsourcing labeling, we study the problem of sequential eliciting information without verification (EIWV) for workers with a heterogeneous and unknown crowd. We propose a reinforcement learning-based approach that is effective against a wide range of settings including potential irrationality and collusion among workers. With the aid of a costly oracle and the inference method, our approach dynamically decides the oracle calls and gains robustness even under the presence of frequent collusion activities. Extensive experiments show the advantage of our approach. Our results also present the first comprehensive experiments of EIWV on large-scale real datasets and the first thorough study of the effects of environmental variables.