Remote Pulse Estimation in the Presence of Face Masks

TL;DR

This paper investigates how face masks affect remote pulse estimation accuracy and proposes a 3D CNN model with data augmentation to improve performance on masked faces, supported by new datasets and evaluations.

Contribution

It introduces a novel 3D CNN approach for masked face pulse estimation, along with datasets and a synthetic mask augmentation method to enhance model robustness.

Findings

Face masks increase heart rate estimation error by over 80%.

Synthetic mask augmentation improves model performance on masked faces.

The proposed method closes the accuracy gap caused by face masks.

Abstract

Remote photoplethysmography (rPPG), a family of techniques for monitoring blood volume changes, may be especially useful for widespread contactless health monitoring using face video from consumer-grade visible-light cameras. The COVID-19 pandemic has caused the widespread use of protective face masks. We found that occlusions from cloth face masks increased the mean absolute error of heart rate estimation by more than 80\% when deploying methods designed on unmasked faces. We show that augmenting unmasked face videos by adding patterned synthetic face masks forces the model to attend to the periocular and forehead regions, improving performance and closing the gap between masked and unmasked pulse estimation. To our knowledge, this paper is the first to analyse the impact of face masks on the accuracy of pulse estimation and offers several novel contributions: (a) 3D CNN-based method designed for remote photoplethysmography in a presence of face masks, (b) two publicly available pulse estimation datasets acquired from 86 unmasked and 61 masked subjects, (c) evaluations of handcrafted algorithms and a 3D CNN trained on videos of unmasked faces and with masks synthetically added, and (d) data augmentation method to add a synthetic mask to a face video.