Integrating Crowdsourcing and Active Learning for Classification of Work-Life Events from Tweets

TL;DR

This paper presents a combined crowdsourcing and active learning approach to efficiently annotate tweets for classifying work-life events, reducing manual effort while maintaining high annotation quality.

Contribution

It introduces a novel pipeline integrating crowdsourcing with active learning strategies to improve annotation efficiency for social media data.

Findings

Crowdsourcing yields high-quality annotations for tweets.

Active learning reduces the number of tweets needed for training.

No significant difference among tested active learning strategies.

Abstract

Social media, especially Twitter, is being increasingly used for research with predictive analytics. In social media studies, natural language processing (NLP) techniques are used in conjunction with expert-based, manual and qualitative analyses. However, social media data are unstructured and must undergo complex manipulation for research use. The manual annotation is the most resource and time-consuming process that multiple expert raters have to reach consensus on every item, but is essential to create gold-standard datasets for training NLP-based machine learning classifiers. To reduce the burden of the manual annotation, yet maintaining its reliability, we devised a crowdsourcing pipeline combined with active learning strategies. We demonstrated its effectiveness through a case study that identifies job loss events from individual tweets. We used Amazon Mechanical Turk platform to recruit annotators from the Internet and designed a number of quality control measures to assure annotation accuracy. We evaluated 4 different active learning strategies (i.e., least confident, entropy, vote entropy, and Kullback-Leibler divergence). The active learning strategies aim at reducing the number of tweets needed to reach a desired performance of automated classification. Results show that crowdsourcing is useful to create high-quality annotations and active learning helps in reducing the number of required tweets, although there was no substantial difference among the strategies tested.