PhySU-Net: Long Temporal Context Transformer for rPPG with Self-Supervised Pre-training

TL;DR

PhySU-Net is a novel transformer-based model for remote photoplethysmography that effectively utilizes long-term temporal context and self-supervised pre-training to enhance performance with limited labeled data.

Contribution

It introduces the first long spatial-temporal map rPPG transformer and a self-supervised pre-training strategy leveraging unlabeled data.

Findings

Superior performance on public datasets (OBF and VIPL-HR).

Self-supervised pre-training improves model accuracy.

Effective long-term temporal modeling in rPPG.

Abstract

Remote photoplethysmography (rPPG) is a promising technology that consists of contactless measuring of cardiac activity from facial videos. Most recent approaches utilize convolutional networks with limited temporal modeling capability or ignore long temporal context. Supervised rPPG methods are also severely limited by scarce data availability. In this work, we propose PhySU-Net, the first long spatial-temporal map rPPG transformer network and a self-supervised pre-training strategy that exploits unlabeled data to improve our model. Our strategy leverages traditional methods and image masking to provide pseudo-labels for self-supervised pre-training. Our model is tested on two public datasets (OBF and VIPL-HR) and shows superior performance in supervised training. Furthermore, we demonstrate that our self-supervised pre-training strategy further improves our model's performance by leveraging representations learned from unlabeled data.