Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning

TL;DR

This paper investigates how the order of instances affects human annotation quality in stream processing systems and proposes an error-avoidance active learning method to improve annotation accuracy in real-time social media analysis.

Contribution

It introduces a human error framework for annotation tasks, demonstrating that sequence order impacts quality, and develops an active learning approach to mitigate annotation errors.

Findings

Sequence order influences annotation accuracy.

Error-avoidance active learning improves label quality.

Method outperforms standard baselines in social media classification.

Abstract

High-quality human annotations are necessary for creating effective machine learning-driven stream processing systems. We study hybrid stream processing systems based on a Human-In-The-Loop Machine Learning (HITL-ML) paradigm, in which one or many human annotators and an automatic classifier (trained at least partially by the human annotators) label an incoming stream of instances. This is typical of many near-real-time social media analytics and web applications, including annotating social media posts during emergencies by digital volunteer groups. From a practical perspective, low-quality human annotations result in wrong labels for retraining automated classifiers and indirectly contribute to the creation of inaccurate classifiers. Considering human annotation as a psychological process allows us to address these limitations. We show that human annotation quality is dependent on the ordering of instances shown to annotators and can be improved by local changes in the instance sequence/order provided to the annotators, yielding a more accurate annotation of the stream. We adapt a theoretically-motivated human error framework of mistakes and slips for the human annotation task to study the effect of ordering instances (i.e., an "annotation schedule"). Further, we propose an error-avoidance approach to the active learning paradigm for stream processing applications robust to these likely human errors (in the form of slips) when deciding a human annotation schedule. We support the human error framework using crowdsourcing experiments and evaluate the proposed algorithm against standard baselines for active learning via extensive experimentation on classification tasks of filtering relevant social media posts during natural disasters.