Focus on Change: Mood Prediction by Learning Emotion Changes via Spatio-Temporal Attention

TL;DR

This paper introduces a novel approach for mood prediction that leverages emotion change information and spatial-temporal attention mechanisms, demonstrating improved performance and generalizability across datasets.

Contribution

It is the first to incorporate emotion change data with spatial-temporal attention modules for mood prediction, enhancing accuracy and robustness.

Findings

Emotion change information improves mood prediction.

Sequential and parallel attention modules enhance performance.

Models trained on AFEW generalize well to EMMA.

Abstract

While emotion and mood interchangeably used, they differ in terms of duration, intensity and attributes. Even as multiple psychology studies examine the mood-emotion relationship, mood prediction has barely been studied. Recent machine learning advances such as the attention mechanism to focus on salient parts of the input data, have only been applied to infer emotions rather than mood. We perform mood prediction by incorporating both mood and emotion change information. We additionally explore spatial and temporal attention, and parallel/sequential arrangements of the spatial and temporal attention modules to improve mood prediction performance. To examine generalizability of the proposed method, we evaluate models trained on the AFEW dataset with EMMA. Experiments reveal that (a) emotion change information is inherently beneficial to mood prediction, and (b) prediction performance improves with the integration of sequential and parallel spatial-temporal attention modules.