DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks

TL;DR

DeepPhys introduces a novel end-to-end deep learning system for accurate, non-contact video-based heart and breathing rate measurement, robust to head rotations and lighting variations, with interpretability via attention visualization.

Contribution

It is the first end-to-end deep convolutional network for physiological measurement from video, incorporating a new motion representation and attention mechanism for robustness.

Findings

Outperforms state-of-the-art methods on RGB and infrared datasets.

Robust to large head rotations and lighting changes.

Provides visualization of physiological signal distributions.

Abstract

Non-contact video-based physiological measurement has many applications in health care and human-computer interaction. Practical applications require measurements to be accurate even in the presence of large head rotations. We propose the first end-to-end system for video-based measurement of heart and breathing rate using a deep convolutional network. The system features a new motion representation based on a skin reflection model and a new attention mechanism using appearance information to guide motion estimation, both of which enable robust measurement under heterogeneous lighting and major motions. Our approach significantly outperforms all current state-of-the-art methods on both RGB and infrared video datasets. Furthermore, it allows spatial-temporal distributions of physiological signals to be visualized via the attention mechanism.