Manipulating emotions for ground truth emotion analysis

TL;DR

This paper explores the use of experimental emotion induction techniques to improve the validity of text-based emotion analysis, revealing limitations of current methods and suggesting directions for future research.

Contribution

It introduces online emotion induction methods into text analysis and evaluates their effectiveness compared to traditional lexicon and classifier approaches.

Findings

Emotion induction procedures successfully created distinct mood states.

Lexicon approaches detected differences but explained limited variance.

Pretrained classifiers performed poorly in identifying true induced emotions.

Abstract

Text data are being used as a lens through which human cognition can be studied at a large scale. Methods like emotion analysis are now in the standard toolkit of computational social scientists but typically rely on third-person annotation with unknown validity. As an alternative, this paper introduces online emotion induction techniques from experimental behavioural research as a method for text-based emotion analysis. Text data were collected from participants who were randomly allocated to a happy, neutral or sad condition. The findings support the mood induction procedure. We then examined how well lexicon approaches can retrieve the induced emotion. All approaches resulted in statistical differences between the true emotion conditions. Overall, only up to one-third of the variance in emotion was captured by text-based measurements. Pretrained classifiers performed poorly on detecting true emotions. The paper concludes with limitations and suggestions for future research.