Crowdsourcing for Beyond Polarity Sentiment Analysis A Pure Emotion Lexicon

TL;DR

This paper introduces a scalable, cost-effective crowdsourcing approach for creating emotion lexicons for beyond polarity sentiment analysis, and evaluates its quality against expert assessments.

Contribution

It presents a novel crowdsourcing method for lexicon creation that bypasses the need for experts or gold standards, and compares crowd and expert evaluations.

Findings

Crowdsourcing effectively produces emotion lexicons for sentiment analysis.

Crowd evaluations are comparable to expert assessments in quality.

The method is scalable and reduces costs for lexicon development.

Abstract

Sentiment analysis aims to uncover emotions conveyed through information. In its simplest form, it is performed on a polarity basis, where the goal is to classify information with positive or negative emotion. Recent research has explored more nuanced ways to capture emotions that go beyond polarity. For these methods to work, they require a critical resource: a lexicon that is appropriate for the task at hand, in terms of the range of emotions it captures diversity. In the past, sentiment analysis lexicons have been created by experts, such as linguists and behavioural scientists, with strict rules. Lexicon evaluation was also performed by experts or gold standards. In our paper, we propose a crowdsourcing method for lexicon acquisition, which is scalable, cost-effective, and doesn't require experts or gold standards. We also compare crowd and expert evaluations of the lexicon, to assess the overall lexicon quality, and the evaluation capabilities of the crowd.