PhysMLE: Generalizable and Priors-Inclusive Multi-task Remote Physiological Measurement

TL;DR

PhysMLE is a novel multi-task learning framework for remote physiological measurement that improves generalization across tasks and incorporates prior physiological knowledge, demonstrated on a new large-scale benchmark and dataset.

Contribution

This paper introduces PhysMLE, a mixture of low-rank experts model with a novel routing mechanism and prior knowledge integration for multi-task rPPG and vital sign measurement.

Findings

PhysMLE outperforms existing methods on the MSSDG benchmark.

The model effectively handles task imbalances and correlations.

A new dataset for multi-task physiological measurement is provided.

Abstract

Remote photoplethysmography (rPPG) has been widely applied to measure heart rate from face videos. To increase the generalizability of the algorithms, domain generalization (DG) attracted increasing attention in rPPG. However, when rPPG is extended to simultaneously measure more vital signs (e.g., respiration and blood oxygen saturation), achieving generalizability brings new challenges. Although partial features shared among different physiological signals can benefit multi-task learning, the sparse and imbalanced target label space brings the seesaw effect over task-specific feature learning. To resolve this problem, we designed an end-to-end Mixture of Low-rank Experts for multi-task remote Physiological measurement (PhysMLE), which is based on multiple low-rank experts with a novel router mechanism, thereby enabling the model to adeptly handle both specifications and correlations within tasks. Additionally, we introduced prior knowledge from physiology among tasks to overcome the imbalance of label space under real-world multi-task physiological measurement. For fair and comprehensive evaluations, this paper proposed a large-scale multi-task generalization benchmark, named Multi-Source Synsemantic Domain Generalization (MSSDG) protocol. Extensive experiments with MSSDG and intra-dataset have shown the effectiveness and efficiency of PhysMLE. In addition, a new dataset was collected and made publicly available to meet the needs of the MSSDG.