Emotions as Ambiguity-aware Ordinal Representations

TL;DR

This paper introduces ambiguity-aware ordinal emotion representations that effectively model the dynamic and ambiguous nature of emotions, outperforming traditional methods on continuous emotion recognition tasks.

Contribution

It presents a novel framework that captures emotion ambiguity and temporal dynamics through ordinal representations, improving continuous emotion prediction accuracy.

Findings

Ordinal representations outperform conventional models on unbounded emotion traces.

They achieve higher CCC and SDA scores, indicating better modeling of emotion dynamics.

Superior SDA performance on bounded traces shows improved relative change capture.

Abstract

Emotions are inherently ambiguous and dynamic phenomena, yet existing continuous emotion recognition approaches either ignore their ambiguity or treat ambiguity as an independent and static variable over time. Motivated by this gap in the literature, in this paper we introduce ambiguity-aware ordinal emotion representations, a novel framework that captures both the ambiguity present in emotion annotation and the inherent temporal dynamics of emotional traces. Specifically, we propose approaches that model emotion ambiguity through its rate of change. We evaluate our framework on two affective corpora -- RECOLA and GameVibe -- testing our proposed approaches on both bounded (arousal, valence) and unbounded (engagement) continuous traces. Our results demonstrate that ordinal representations outperform conventional ambiguity-aware models on unbounded labels, achieving the highest Concordance Correlation Coefficient (CCC) and Signed Differential Agreement (SDA) scores, highlighting their effectiveness in modeling the traces' dynamics. For bounded traces, ordinal representations excel in SDA, revealing their superior ability to capture relative changes of annotated emotion traces.