WASSA-2017 Shared Task on Emotion Intensity

TL;DR

This paper introduces a shared task on detecting emotion intensity in tweets, providing annotated datasets and benchmarking machine learning approaches to advance understanding of emotional expression in social media.

Contribution

It presents the first dataset and shared task for emotion intensity detection in tweets, using best--worst scaling for annotation and evaluating multiple systems.

Findings

Best system achieved a Pearson correlation of 0.747 with gold scores.

Annotated datasets enable reliable fine-grained emotion intensity rankings.

The shared task fosters progress in computational emotion analysis.

Abstract

We present the first shared task on detecting the intensity of emotion felt by the speaker of a tweet. We create the first datasets of tweets annotated for anger, fear, joy, and sadness intensities using a technique called best--worst scaling (BWS). We show that the annotations lead to reliable fine-grained intensity scores (rankings of tweets by intensity). The data was partitioned into training, development, and test sets for the competition. Twenty-two teams participated in the shared task, with the best system obtaining a Pearson correlation of 0.747 with the gold intensity scores. We summarize the machine learning setups, resources, and tools used by the participating teams, with a focus on the techniques and resources that are particularly useful for the task. The emotion intensity dataset and the shared task are helping improve our understanding of how we convey more or less intense emotions through language.