Automatic Detection of Sentimentality from Facial Expressions

TL;DR

This paper introduces a novel approach for detecting sentimentality, a complex emotional state, from facial expressions using a new dataset, weak labeling, and machine learning, marking the first effort in this area.

Contribution

It presents the first methodology for sentimentality detection from facial cues, including dataset creation, weak labeling strategy, and a neural network-based detection model.

Findings

Promising accuracy in sentimentality detection

Effective use of Action Units for weak labeling

New ad-level metrics for model evaluation

Abstract

Emotion recognition has received considerable attention from the Computer Vision community in the last 20 years. However, most of the research focused on analyzing the six basic emotions (e.g. joy, anger, surprise), with a limited work directed to other affective states. In this paper, we tackle sentimentality (strong feeling of heartwarming or nostalgia), a new emotional state that has few works in the literature, and no guideline defining its facial markers. To this end, we first collect a dataset of 4.9K videos of participants watching some sentimental and non-sentimental ads, and then we label the moments evoking sentimentality in the ads. Second, we use the ad-level labels and the facial Action Units (AUs) activation across different frames for defining some weak frame-level sentimentality labels. Third, we train a Multilayer Perceptron (MLP) using the AUs activation for sentimentality detection. Finally, we define two new ad-level metrics for evaluating our model performance. Quantitative and qualitative results show promising results for sentimentality detection. To the best of our knowledge this is the first work to address the problem of sentimentality detection.