PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer

TL;DR

This paper introduces PhysFormer++: a novel video transformer architecture for contactless physiological measurement from facial videos, leveraging long-range spatio-temporal features and advanced training strategies to outperform existing methods.

Contribution

The paper proposes PhysFormer and PhysFormer++, innovative transformer-based models that effectively capture long-range spatio-temporal features for rPPG, with training techniques that reduce overfitting and eliminate the need for large-scale pretraining.

Findings

Superior performance on four benchmark datasets

Effective long-range spatio-temporal feature modeling

Can be trained from scratch without large pretraining datasets

Abstract

Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.