A Weakly Supervised Approach to Emotion-change Prediction and Improved Mood Inference

TL;DR

This paper introduces a weakly supervised method that leverages emotion-change information, derived without explicit labels, to improve mood prediction from long video clips, highlighting the importance of mood-emotion interplay.

Contribution

It proposes a novel approach to incorporate emotion-change signals into mood inference without annotated labels, enhancing mood prediction accuracy.

Findings

Emotion-change information improves mood prediction

Multimodal models outperform unimodal ones

Emotion-change labels are generated via metric learning

Abstract

Whilst a majority of affective computing research focuses on inferring emotions, examining mood or understanding the \textit{mood-emotion interplay} has received significantly less attention. Building on prior work, we (a) deduce and incorporate emotion-change ($\Delta$) information for inferring mood, without resorting to annotated labels, and (b) attempt mood prediction for long duration video clips, in alignment with the characterisation of mood. We generate the emotion-change ($\Delta$) labels via metric learning from a pre-trained Siamese Network, and use these in addition to mood labels for mood classification. Experiments evaluating \textit{unimodal} (training only using mood labels) vs \textit{multimodal} (training using mood plus $\Delta$ labels) models show that mood prediction benefits from the incorporation of emotion-change information, emphasising the importance of modelling the mood-emotion interplay for effective mood inference.