Causal affect prediction model using a facial image sequence

TL;DR

This paper introduces CAPNet, a causal affect prediction model that uses only past facial images to accurately predict affective states, enabling real-time emotion analysis without future data.

Contribution

The paper presents a novel causal inference-based neural network that predicts affective valence and arousal solely from past facial images, suitable for real-time applications.

Findings

CAPNet outperforms baseline models in the ABAW2 challenge.

CAPNet predicts affective states with only past data, enabling real-time use.

Experimental results demonstrate reliable affect prediction one-third of a second earlier.

Abstract

Among human affective behavior research, facial expression recognition research is improving in performance along with the development of deep learning. However, for improved performance, not only past images but also future images should be used along with corresponding facial images, but there are obstacles to the application of this technique to real-time environments. In this paper, we propose the causal affect prediction network (CAPNet), which uses only past facial images to predict corresponding affective valence and arousal. We train CAPNet to learn causal inference between past images and corresponding affective valence and arousal through supervised learning by pairing the sequence of past images with the current label using the Aff-Wild2 dataset. We show through experiments that the well-trained CAPNet outperforms the baseline of the second challenge of the Affective Behavior Analysis in-the-wild (ABAW2) Competition by predicting affective valence and arousal only with past facial images one-third of a second earlier. Therefore, in real-time application, CAPNet can reliably predict affective valence and arousal only with past data.