# Modeling Human Annotation Errors to Design Bias-Aware Systems for Social Stream Processing

## TL;DR

This paper investigates how the order of data presentation affects human annotation quality in social media analysis, proposing an error-mitigating active learning approach to improve accuracy and reduce bias in real-time social media classification.

## Contribution

It introduces a novel annotation schedule optimization method and an active learning algorithm that accounts for human errors, enhancing annotation quality for social media analytics.

## Key findings

- Order of data presentation impacts annotation accuracy.
- The proposed algorithm improves classification performance.
- Considering annotation order reduces bias in social media analysis.

## Abstract

High-quality human annotations are necessary to create effective machine learning systems for social media. Low-quality human annotations indirectly contribute to the creation of inaccurate or biased learning systems. We show that human annotation quality is dependent on the ordering of instances shown to annotators (referred as 'annotation schedule'), and can be improved by local changes in the instance ordering provided to the annotators, yielding a more accurate annotation of the data stream for efficient real-time social media analytics. We propose an error-mitigating active learning algorithm that is robust with respect to some cases of human errors when deciding an annotation schedule. We validate the human error model and evaluate the proposed algorithm against strong baselines by experimenting on classification tasks of relevant social media posts during crises. According to these experiments, considering the order in which data instances are presented to human annotators leads to both an increase in accuracy for machine learning and awareness toward some potential biases in human learning that may affect the automated classifier.