Emotion Ratings: How Intensity, Annotation Confidence and Agreements are Entangled

TL;DR

This study investigates how human confidence in emotion annotation correlates with agreement levels and emotion intensity, revealing that confidence can predict annotation disagreement and is linked to perceived emotional strength.

Contribution

The paper demonstrates that human confidence scores are indicative of annotation disagreements and are correlated with emotion intensity, offering new insights for emotion annotation and modeling.

Findings

Confidence approximates inter-annotator disagreement.

Confidence correlates with emotion intensity.

Stronger affect leads to higher confidence in annotations.

Abstract

When humans judge the affective content of texts, they also implicitly assess the correctness of such judgment, that is, their confidence. We hypothesize that people's (in)confidence that they performed well in an annotation task leads to (dis)agreements among each other. If this is true, confidence may serve as a diagnostic tool for systematic differences in annotations. To probe our assumption, we conduct a study on a subset of the Corpus of Contemporary American English, in which we ask raters to distinguish neutral sentences from emotion-bearing ones, while scoring the confidence of their answers. Confidence turns out to approximate inter-annotator disagreements. Further, we find that confidence is correlated to emotion intensity: perceiving stronger affect in text prompts annotators to more certain classification performances. This insight is relevant for modelling studies of intensity, as it opens the question wether automatic regressors or classifiers actually predict intensity, or rather human's self-perceived confidence.