Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective

TL;DR

This paper investigates how face masks impact emotion recognition and demonstrates that using full body cues with deep learning improves accuracy over facial cues alone, especially under mask occlusion.

Contribution

It introduces a multi-modal deep learning framework that combines facial and body features for emotion recognition, effectively addressing mask-induced challenges.

Findings

Full body input outperforms masked face alone in emotion recognition.

Separate processing and fusion of facial and body features enhance accuracy.

Temporal modeling improves emotion recognition performance.

Abstract

The COVID-19 pandemic has undoubtedly changed the standards and affected all aspects of our lives, especially social communication. It has forced people to extensively wear medical face masks, in order to prevent transmission. This face occlusion can strongly irritate emotional reading from the face and urges us to incorporate the whole body as an emotional cue. In this paper, we conduct insightful studies about the effect of face occlusion on emotion recognition performance, and showcase the superiority of full body input over the plain masked face. We utilize a deep learning model based on the Temporal Segment Network framework, and aspire to fully overcome the face mask consequences. Although facial and bodily features can be learned from a single input, this may lead to irrelevant information confusion. By processing those features separately and fusing their prediction scores, we are more effectively taking advantage of both modalities. This framework also naturally supports temporal modeling, by mingling information among neighboring frames. In combination, these techniques form an effective system capable of tackling emotion recognition difficulties, caused by safety protocols applied in crucial areas.