Attending to Emotional Narratives

TL;DR

This paper demonstrates that attention mechanisms like Transformers and Memory Fusion Networks effectively improve multimodal time-series emotion recognition, achieving performance comparable to human raters on autobiographical narratives.

Contribution

It adapts and applies advanced attention-based models to emotion recognition in multimodal narratives, showing their effectiveness and potential in affective computing.

Findings

Models perform extremely well, sometimes matching human ratings.

Attention mechanisms generalize effectively to multimodal emotion recognition.

The approach offers promising implications for affective computing.

Abstract

Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalities and time---also generalize well to multimodal time-series emotion recognition. Using a recently-introduced dataset of emotional autobiographical narratives, we adapt and apply these two attention mechanisms to predict emotional valence over time. Our models perform extremely well, in some cases reaching a performance comparable with human raters. We end with a discussion of the implications of attention mechanisms to affective computing.