Implicit Design Choices and Their Impact on Emotion Recognition Model Development and Evaluation

TL;DR

This paper investigates how implicit design choices in emotion recognition models affect their development and evaluation, emphasizing dataset diversity, label subjectivity, confounding variables, and demographic privacy.

Contribution

It introduces a comprehensive analysis of implicit design factors and proposes methods to improve robustness, fairness, and real-world applicability of emotion recognition models.

Findings

Collected the Multimodal Stressed Emotion dataset with controlled stressors.

Analyzed the impact of data augmentation and annotation schemes on emotion perception.

Used adversarial networks to remove stress and demographic information from representations.

Abstract

Emotion recognition is a complex task due to the inherent subjectivity in both the perception and production of emotions. The subjectivity of emotions poses significant challenges in developing accurate and robust computational models. This thesis examines critical facets of emotion recognition, beginning with the collection of diverse datasets that account for psychological factors in emotion production. To handle the challenge of non-representative training data, this work collects the Multimodal Stressed Emotion dataset, which introduces controlled stressors during data collection to better represent real-world influences on emotion production. To address issues with label subjectivity, this research comprehensively analyzes how data augmentation techniques and annotation schemes impact emotion perception and annotator labels. It further handles natural confounding variables and variations by employing adversarial networks to isolate key factors like stress from learned emotion representations during model training. For tackling concerns about leakage of sensitive demographic variables, this work leverages adversarial learning to strip sensitive demographic information from multimodal encodings. Additionally, it proposes optimized sociological evaluation metrics aligned with cost-effective, real-world needs for model testing. This research advances robust, practical emotion recognition through multifaceted studies of challenges in datasets, labels, modeling, demographic and membership variable encoding in representations, and evaluation. The groundwork has been laid for cost-effective, generalizable emotion recognition models that are less likely to encode sensitive demographic information.