Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset

TL;DR

This paper introduces the Stanford Emotional Narratives Dataset (SENDv1), a multimodal, naturalistic video dataset for modeling dynamic emotions over time, and evaluates several state-of-the-art models on it.

Contribution

The paper presents the first version of SEND, a challenging new dataset for time-series emotion recognition in complex, naturalistic stories, and benchmarks models against human performance.

Findings

Models like LSTM and Variational RNN perform comparably to humans on SEND.

SEND provides a rich, multimodal resource for advancing affective computing research.

Contemporary models face challenges capturing emotions in complex narratives.

Abstract

Human emotions unfold over time, and more affective computing research has to prioritize capturing this crucial component of real-world affect. Modeling dynamic emotional stimuli requires solving the twin challenges of time-series modeling and of collecting high-quality time-series datasets. We begin by assessing the state-of-the-art in time-series emotion recognition, and we review contemporary time-series approaches in affective computing, including discriminative and generative models. We then introduce the first version of the Stanford Emotional Narratives Dataset (SENDv1): a set of rich, multimodal videos of self-paced, unscripted emotional narratives, annotated for emotional valence over time. The complex narratives and naturalistic expressions in this dataset provide a challenging test for contemporary time-series emotion recognition models. We demonstrate several baseline and state-of-the-art modeling approaches on the SEND, including a Long Short-Term Memory model and a multimodal Variational Recurrent Neural Network, which perform comparably to the human-benchmark. We end by discussing the implications for future research in time-series affective computing.