Non-contact PPG Signal and Heart Rate Estimation with Multi-hierarchical Convolutional Network

TL;DR

This paper introduces a multi-hierarchical spatio-temporal convolutional network that accurately estimates remote heart rate from face videos by capturing facial color and motion features, outperforming existing methods.

Contribution

The study proposes a novel multi-hierarchical convolutional network with modules for facial feature extraction, spatio-temporal correlation, and adaptive ROI generation for improved remote HR estimation.

Findings

Achieved lower MAE on UBFC-RPPG dataset (2.15 bpm)

Outperformed state-of-the-art methods on three datasets

Effectively captured tiny facial motions for HR estimation

Abstract

Heartbeat rhythm and heart rate (HR) are important physiological parameters of the human body. This study presents an efficient multi-hierarchical spatio-temporal convolutional network that can quickly estimate remote physiological (rPPG) signal and HR from face video clips. First, the facial color distribution characteristics are extracted using a low-level face feature generation (LFFG) module. Then, the three-dimensional (3D) spatio-temporal stack convolution module (STSC) and multi-hierarchical feature fusion module (MHFF) are used to strengthen the spatio-temporal correlation of multi-channel features. In the MHFF, sparse optical flow is used to capture the tiny motion information of faces between frames and generate a self-adaptive region of interest (ROI) skin mask. Finally, the signal prediction module (SP) is used to extract the estimated rPPG signal. The heart rate estimation results show that the proposed network overperforms the state-of-the-art methods on three datasets, 1) UBFC-RPPG, 2) COHFACE, 3) our dataset, with the mean absolute error (MAE) of 2.15, 5.57, 1.75 beats per minute (bpm) respectively.