A Minimax Optimal Algorithm for Crowdsourcing

TL;DR

This paper introduces Triangular Estimation (TE), a novel, efficient, and minimax optimal algorithm for estimating worker reliability in crowdsourcing, validated through theoretical bounds and empirical tests.

Contribution

The paper presents a new lower bound on estimation error and proposes TE, a non-iterative, streaming-compatible algorithm that achieves minimax optimality in crowdsourcing reliability estimation.

Findings

TE is minimax optimal and matches the theoretical lower bound.

TE performs well on both synthetic and real-world datasets.

The algorithm is computationally efficient and suitable for real-time applications.

Abstract

We consider the problem of accurately estimating the reliability of workers based on noisy labels they provide, which is a fundamental question in crowdsourcing. We propose a novel lower bound on the minimax estimation error which applies to any estimation procedure. We further propose Triangular Estimation (TE), an algorithm for estimating the reliability of workers. TE has low complexity, may be implemented in a streaming setting when labels are provided by workers in real time, and does not rely on an iterative procedure. We further prove that TE is minimax optimal and matches our lower bound. We conclude by assessing the performance of TE and other state-of-the-art algorithms on both synthetic and real-world data sets.