Learning to Predict the Wisdom of Crowds

TL;DR

This paper introduces CrowdSense, an online algorithm that efficiently estimates the crowd's majority opinion by selectively sampling labelers, balancing exploration and exploitation to optimize crowdsourcing accuracy.

Contribution

The paper presents a novel online algorithm, CrowdSense, for approximating crowd opinions with limited queries by dynamically selecting and weighting labelers.

Findings

CrowdSense effectively approximates crowd opinions with fewer queries.

The algorithm balances exploration and exploitation for optimal labeler selection.

Results demonstrate improved accuracy over baseline methods.

Abstract

The problem of "approximating the crowd" is that of estimating the crowd's majority opinion by querying only a subset of it. Algorithms that approximate the crowd can intelligently stretch a limited budget for a crowdsourcing task. We present an algorithm, "CrowdSense," that works in an online fashion to dynamically sample subsets of labelers based on an exploration/exploitation criterion. The algorithm produces a weighted combination of a subset of the labelers' votes that approximates the crowd's opinion.