Globally Optimal Crowdsourcing Quality Management

TL;DR

This paper develops algorithms that find globally optimal solutions for crowdsourcing quality management, improving accuracy over EM-based methods by considering all possible task answer mappings efficiently.

Contribution

It introduces algorithms that guarantee global optimality in estimating true answers and worker quality, overcoming EM's local optima limitations.

Findings

Algorithms often outperform EM-based methods in accuracy.

Significant reduction in the number of mappings considered, enabling practical computation.

Provides complexity characterization of the global optimal algorithms.

Abstract

We study crowdsourcing quality management, that is, given worker responses to a set of tasks, our goal is to jointly estimate the true answers for the tasks, as well as the quality of the workers. Prior work on this problem relies primarily on applying Expectation-Maximization (EM) on the underlying maximum likelihood problem to estimate true answers as well as worker quality. Unfortunately, EM only provides a locally optimal solution rather than a globally optimal one. Other solutions to the problem (that do not leverage EM) fail to provide global optimality guarantees as well. In this paper, we focus on filtering, where tasks require the evaluation of a yes/no predicate, and rating, where tasks elicit integer scores from a finite domain. We design algorithms for finding the global optimal estimates of correct task answers and worker quality for the underlying maximum likelihood problem, and characterize the complexity of these algorithms. Our algorithms conceptually consider all mappings from tasks to true answers (typically a very large number), leveraging two key ideas to reduce, by several orders of magnitude, the number of mappings under consideration, while preserving optimality. We also demonstrate that these algorithms often find more accurate estimates than EM-based algorithms. This paper makes an important contribution towards understanding the inherent complexity of globally optimal crowdsourcing quality management.