Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

TL;DR

This paper explores deep learning architectures, specifically CNNs and CNN-RNN hybrids, for continuous emotion recognition from videos, demonstrating their effectiveness in capturing spatiotemporal features and achieving state-of-the-art results.

Contribution

It introduces and evaluates novel CNN-based models, including inflated 3D-CNNs and convolutional recurrent networks, for improved continuous emotion recognition in-the-wild.

Findings

CNN architectures effectively encode spatiotemporal information.

Models achieve state-of-the-art results on SEWA-DB dataset.

Fine-tuning pre-trained models enhances performance.

Abstract

Facial expressions are one of the most powerful ways for depicting specific patterns in human behavior and describing human emotional state. Despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot handle properly variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task even for humans. In this paper, we investigate the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) for continuous emotion recognition using long video sequences captured in-the-wild. This study focuses on deep learning models that allow encoding spatiotemporal relations in videos considering a complex and multi-dimensional emotion space, where values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks combining 2D-CNNs and long short term-memory units, and inflated 3D-CNN models, which are built by inflating the weights of a pre-trained 2D-CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode the spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.