Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wild

TL;DR

This paper introduces a spatio-temporal Transformer model that effectively captures long-range dependencies in videos for dynamic facial expression recognition, outperforming CNN-based methods.

Contribution

The paper proposes a novel unified Transformer architecture with spatial and temporal attention for in-the-wild facial expression recognition, incorporating a new loss function for better feature discrimination.

Findings

Outperforms CNN-based methods on DFEW and AFEW datasets

Effectively models long-range dependencies in videos

Achieves higher accuracy in dynamic facial expression recognition

Abstract

Previous methods for dynamic facial expression in the wild are mainly based on Convolutional Neural Networks (CNNs), whose local operations ignore the long-range dependencies in videos. To solve this problem, we propose the spatio-temporal Transformer (STT) to capture discriminative features within each frame and model contextual relationships among frames. Spatio-temporal dependencies are captured and integrated by our unified Transformer. Specifically, given an image sequence consisting of multiple frames as input, we utilize the CNN backbone to translate each frame into a visual feature sequence. Subsequently, the spatial attention and the temporal attention within each block are jointly applied for learning spatio-temporal representations at the sequence level. In addition, we propose the compact softmax cross entropy loss to further encourage the learned features have the minimum intra-class distance and the maximum inter-class distance. Experiments on two in-the-wild dynamic facial expression datasets (i.e., DFEW and AFEW) indicate that our method provides an effective way to make use of the spatial and temporal dependencies for dynamic facial expression recognition. The source code and the training logs will be made publicly available.