Fundamental Limits of Budget-Fidelity Trade-off in Label Crowdsourcing

TL;DR

This paper models and analyzes the fundamental limits of accuracy in crowdsourcing tasks using an information theoretic framework, revealing the maximum achievable fidelity given budget constraints and worker imperfections.

Contribution

It introduces a rate-distortion framework for crowdsourcing, characterizes the fundamental fidelity limits, and proposes a $k$-ary incidence coding scheme for optimized querying.

Findings

Identifies the maximum achievable fidelity in crowdsourcing under budget constraints.

Develops a joint source-channel coding scheme for human-in-the-loop inference.

Proposes and analyzes the $k$-ary incidence coding query scheme.

Abstract

Digital crowdsourcing (CS) is a modern approach to perform certain large projects using small contributions of a large crowd. In CS, a taskmaster typically breaks down the project into small batches of tasks and assigns them to so-called workers with imperfect skill levels. The crowdsourcer then collects and analyzes the results for inference and serving the purpose of the project. In this work, the CS problem, as a human-in-the-loop computation problem, is modeled and analyzed in an information theoretic rate-distortion framework. The purpose is to identify the ultimate fidelity that one can achieve by any form of query from the crowd and any decoding (inference) algorithm with a given budget. The results are established by a joint source channel (de)coding scheme, which represent the query scheme and inference, over parallel noisy channels, which model workers with imperfect skill levels. We also present and analyze a query scheme dubbed $k$-ary incidence coding and study optimized query pricing in this setting.