Streaming Bayesian Inference for Crowdsourced Classification

TL;DR

This paper introduces SBIC, a streaming Bayesian algorithm for crowdsourced binary classification that is computationally efficient, accurate, and provides theoretical guarantees in real-time and offline scenarios.

Contribution

The paper presents SBIC, a novel streaming Bayesian inference algorithm that overcomes computational and theoretical limitations of existing crowdsourcing methods.

Findings

SBIC achieves accuracy comparable to state-of-the-art algorithms.

SBIC operates efficiently in real-time online settings.

SBIC has provable asymptotic guarantees in various settings.

Abstract

A key challenge in crowdsourcing is inferring the ground truth from noisy and unreliable data. To do so, existing approaches rely on collecting redundant information from the crowd, and aggregating it with some probabilistic method. However, oftentimes such methods are computationally inefficient, are restricted to some specific settings, or lack theoretical guarantees. In this paper, we revisit the problem of binary classification from crowdsourced data. Specifically we propose Streaming Bayesian Inference for Crowdsourcing (SBIC), a new algorithm that does not suffer from any of these limitations. First, SBIC has low complexity and can be used in a real-time online setting. Second, SBIC has the same accuracy as the best state-of-the-art algorithms in all settings. Third, SBIC has provable asymptotic guarantees both in the online and offline settings.