The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video

TL;DR

This paper introduces a fully self-supervised contrastive learning method for remote photoplethysmography (rPPG) that estimates blood volume changes from facial videos without needing labeled data, achieving competitive results.

Contribution

It presents a novel self-supervised contrastive learning approach for rPPG that eliminates reliance on expensive ground truth labels and incorporates a learned saliency resampling module for interpretability.

Findings

Comparable or better performance than supervised methods

Effective use of saliency resampling for interpretability

No reliance on annotated physiological data

Abstract

The ability to reliably estimate physiological signals from video is a powerful tool in low-cost, pre-clinical health monitoring. In this work we propose a new approach to remote photoplethysmography (rPPG) - the measurement of blood volume changes from observations of a person's face or skin. Similar to current state-of-the-art methods for rPPG, we apply neural networks to learn deep representations with invariance to nuisance image variation. In contrast to such methods, we employ a fully self-supervised training approach, which has no reliance on expensive ground truth physiological training data. Our proposed method uses contrastive learning with a weak prior over the frequency and temporal smoothness of the target signal of interest. We evaluate our approach on four rPPG datasets, showing that comparable or better results can be achieved compared to recent supervised deep learning methods but without using any annotation. In addition, we incorporate a learned saliency resampling module into both our unsupervised approach and supervised baseline. We show that by allowing the model to learn where to sample the input image, we can reduce the need for hand-engineered features while providing some interpretability into the model's behavior and possible failure modes. We release code for our complete training and evaluation pipeline to encourage reproducible progress in this exciting new direction.