Crowd-Machine Collaboration for Item Screening

TL;DR

This paper presents hybrid algorithms combining crowd and machine classifiers to efficiently screen items based on predicates, demonstrating improved performance and cost-effectiveness over single-method approaches.

Contribution

It introduces novel hybrid algorithms for item screening that leverage both crowd and machine classifiers, applicable across various domains.

Findings

Hybrid algorithms outperform human-only screening in efficiency.

Hybrid algorithms reduce costs compared to machine-only screening.

Performance gains are demonstrated across multiple domains.

Abstract

In this paper we describe how crowd and machine classifier can be efficiently combined to screen items that satisfy a set of predicates. We show that this is a recurring problem in many domains, present machine-human (hybrid) algorithms that screen items efficiently and estimate the gain over human-only or machine-only screening in terms of performance and cost.