Joint Spatial-Temporal Modeling and Contrastive Learning for Self-supervised Heart Rate Measurement

TL;DR

This paper presents two innovative self-supervised learning methods for heart rate estimation from facial videos, combining spatial-temporal modeling and contrastive learning, achieving high accuracy and securing second place in a challenge.

Contribution

The paper introduces two novel self-supervised solutions for HR measurement that integrate spatial-temporal modeling and contrastive learning, and combines them for improved accuracy.

Findings

Achieved an RMSE of 8.85277 on the test dataset.

Secured 2nd place in the RePSS Challenge Track 1.

Demonstrated effectiveness of combined self-supervised approaches.

Abstract

This paper briefly introduces the solutions developed by our team, HFUT-VUT, for Track 1 of self-supervised heart rate measurement in the 3rd Vision-based Remote Physiological Signal Sensing (RePSS) Challenge hosted at IJCAI 2024. The goal is to develop a self-supervised learning algorithm for heart rate (HR) estimation using unlabeled facial videos. To tackle this task, we present two self-supervised HR estimation solutions that integrate spatial-temporal modeling and contrastive learning, respectively. Specifically, we first propose a non-end-to-end self-supervised HR measurement framework based on spatial-temporal modeling, which can effectively capture subtle rPPG clues and leverage the inherent bandwidth and periodicity characteristics of rPPG to constrain the model. Meanwhile, we employ an excellent end-to-end solution based on contrastive learning, aiming to generalize across different scenarios from complementary perspectives. Finally, we combine the strengths of the above solutions through an ensemble strategy to generate the final predictions, leading to a more accurate HR estimation. As a result, our solutions achieved a remarkable RMSE score of 8.85277 on the test dataset, securing \textbf{2nd place} in Track 1 of the challenge.