Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling

TL;DR

This paper introduces a reliable method using Best-Worst Scaling to obtain consistent fine-grained sentiment scores for words across multiple languages and domains, revealing the smallest perceptible sentiment difference.

Contribution

It applies Best-Worst Scaling to derive stable sentiment association scores and identifies the minimal perceptible sentiment difference for native speakers.

Findings

Sentiment rankings are consistent across different annotators.

Best-Worst Scaling effectively captures fine-grained sentiment scores.

Minimum perceptible sentiment difference is established for native speakers.

Abstract

Access to word-sentiment associations is useful for many applications, including sentiment analysis, stance detection, and linguistic analysis. However, manually assigning fine-grained sentiment association scores to words has many challenges with respect to keeping annotations consistent. We apply the annotation technique of Best-Worst Scaling to obtain real-valued sentiment association scores for words and phrases in three different domains: general English, English Twitter, and Arabic Twitter. We show that on all three domains the ranking of words by sentiment remains remarkably consistent even when the annotation process is repeated with a different set of annotators. We also, for the first time, determine the minimum difference in sentiment association that is perceptible to native speakers of a language.