Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation

TL;DR

This paper introduces a novel orientation-conditioned facial texture video representation leveraging 3D facial surface modeling to improve the robustness of remote photoplethysmography (rPPG) estimation in videos with dynamic, unconstrained motion, achieving significant performance gains.

Contribution

The study presents a new 3D facial surface-based video representation that enhances motion robustness in video-based facial rPPG estimation, outperforming existing methods in challenging scenarios.

Findings

18.2% performance improvement in cross-dataset testing on MMPD

Up to 29.6% performance increase across various motion scenarios

Effective disentangling of motion through 3D facial surface modeling

Abstract

Camera-based remote photoplethysmography (rPPG) enables contactless measurement of important physiological signals such as pulse rate (PR). However, dynamic and unconstrained subject motion introduces significant variability into the facial appearance in video, confounding the ability of video-based methods to accurately extract the rPPG signal. In this study, we leverage the 3D facial surface to construct a novel orientation-conditioned facial texture video representation which improves the motion robustness of existing video-based facial rPPG estimation methods. Our proposed method achieves a significant 18.2% performance improvement in cross-dataset testing on MMPD over our baseline using the PhysNet model trained on PURE, highlighting the efficacy and generalization benefits of our designed video representation. We demonstrate significant performance improvements of up to 29.6% in all tested motion scenarios in cross-dataset testing on MMPD, even in the presence of dynamic and unconstrained subject motion, emphasizing the benefits of disentangling motion through modeling the 3D facial surface for motion robust facial rPPG estimation. We validate the efficacy of our design decisions and the impact of different video processing steps through an ablation study. Our findings illustrate the potential strengths of exploiting the 3D facial surface as a general strategy for addressing dynamic and unconstrained subject motion in videos. The code is available at https://samcantrill.github.io/orientation-uv-rppg/.