Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition

TL;DR

This paper introduces a softmax-based temporal pooling and weighted sampling strategy for 3D CNNs, enhancing facial expression recognition accuracy and efficiency by focusing on the most relevant video clips during training and inference.

Contribution

It proposes a novel softmax temporal pooling method combined with weighted sampling to improve 3D CNN training for facial expression recognition.

Findings

Improved recognition accuracy on benchmark datasets.

Reduced computational cost during training and inference.

Enhanced robustness to video trimming inaccuracies.

Abstract

Training deep learning models for accurate spatiotemporal recognition of facial expressions in videos requires significant computational resources. For practical reasons, 3D Convolutional Neural Networks (3D CNNs) are usually trained with relatively short clips randomly extracted from videos. However, such uniform sampling is generally sub-optimal because equal importance is assigned to each temporal clip. In this paper, we present a strategy for efficient video-based training of 3D CNNs. It relies on softmax temporal pooling and a weighted sampling mechanism to select the most relevant training clips. The proposed softmax strategy provides several advantages: a reduced computational complexity due to efficient clip sampling, and an improved accuracy since temporal weighting focuses on more relevant clips during both training and inference. Experimental results obtained with the proposed method on several facial expression recognition benchmarks show the benefits of focusing on more informative clips in training videos. In particular, our approach improves performance and computational cost by reducing the impact of inaccurate trimming and coarse annotation of videos, and heterogeneous distribution of visual information across time.