Facial Expression Recognition Under Partial Occlusion from Virtual Reality Headsets based on Transfer Learning

TL;DR

This paper addresses facial expression recognition under severe occlusion caused by VR headsets by using transfer learning on modified datasets, achieving results comparable to existing methods.

Contribution

It introduces a geometric occlusion model for VR headsets and applies transfer learning with VGG and ResNet to improve FER under occlusion.

Findings

Achieves comparable accuracy to existing FER methods under VR headset occlusion.

Develops a realistic occlusion simulation model for FER datasets.

Demonstrates effectiveness of transfer learning in occluded FER scenarios.

Abstract

Facial expressions of emotion are a major channel in our daily communications, and it has been subject of intense research in recent years. To automatically infer facial expressions, convolutional neural network based approaches has become widely adopted due to their proven applicability to Facial Expression Recognition (FER) task.On the other hand Virtual Reality (VR) has gained popularity as an immersive multimedia platform, where FER can provide enriched media experiences. However, recognizing facial expression while wearing a head-mounted VR headset is a challenging task due to the upper half of the face being completely occluded. In this paper we attempt to overcome these issues and focus on facial expression recognition in presence of a severe occlusion where the user is wearing a head-mounted display in a VR setting. We propose a geometric model to simulate occlusion resulting from a Samsung Gear VR headset that can be applied to existing FER datasets. Then, we adopt a transfer learning approach, starting from two pretrained networks, namely VGG and ResNet. We further fine-tune the networks on FER+ and RAF-DB datasets. Experimental results show that our approach achieves comparable results to existing methods while training on three modified benchmark datasets that adhere to realistic occlusion resulting from wearing a commodity VR headset. Code for this paper is available at: https://github.com/bita-github/MRP-FER