Implicit Media Tagging and Affect Prediction from video of spontaneous facial expressions, recorded with depth camera

TL;DR

This paper introduces a method to automatically evaluate emotional responses from spontaneous facial expressions captured by a depth camera, useful for media tagging and human-computer interaction, by predicting a 4-dimensional affective state.

Contribution

It constructs a new database of emotionally evocative videos with affect labels and develops a learning-based two-step prediction method for affect from facial activity.

Findings

Successfully predicts Valence, Arousal, Likability, Rewatch scales

Identifies peak emotional response periods in recordings

High agreement between independent viewers' responses

Abstract

We present a method that automatically evaluates emotional response from spontaneous facial activity recorded by a depth camera. The automatic evaluation of emotional response, or affect, is a fascinating challenge with many applications, including human-computer interaction, media tagging and human affect prediction. Our approach in addressing this problem is based on the inferred activity of facial muscles over time, as captured by a depth camera recording an individual's facial activity. Our contribution is two-fold: First, we constructed a database of publicly available short video clips, which elicit a strong emotional response in a consistent manner across different individuals. Each video was tagged by its characteristic emotional response along 4 scales: \emph{Valence, Arousal, Likability} and \emph{Rewatch} (the desire to watch again). The second contribution is a two-step prediction method, based on learning, which was trained and tested using this database of tagged video clips. Our method was able to successfully predict the aforementioned 4 dimensional representation of affect, as well as to identify the period of strongest emotional response in the viewing recordings, in a method that is blind to the video clip being watch, revealing a significantly high agreement between the recordings of independent viewers.