Engineering Crowdsourced Stream Processing Systems

TL;DR

This paper introduces a framework for designing and evaluating crowdsourced stream processing systems, demonstrating improved accuracy and efficiency in classifying social media during crises.

Contribution

It presents a taxonomy, design principles, evaluation metrics, and patterns for CSP systems, along with a case study applying these to a real-world system.

Findings

AIDR achieves higher classification accuracy than pure stream processing.

AIDR reduces manual effort compared to pure crowdsourcing.

The framework effectively guides CSP system design and analysis.

Abstract

A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort.