Combining Facial Videos and Biosignals for Stress Estimation During Driving

TL;DR

This paper presents a multimodal framework combining facial videos and physiological signals for stress estimation during driving, demonstrating significant performance improvements and potential generalization to other contexts.

Contribution

It introduces a novel dense 3D facial representation and a Transformer-based temporal model, with cross-modal attention fusion enhancing stress detection accuracy.

Findings

Facial components show consistent stress responses comparable to physiological markers.

Cross-modal attention fusion improves AUROC from 52.7% to 92.0%.

The framework generalizes beyond driving scenarios.

Abstract

Reliable stress recognition is critical in applications such as medical monitoring and safety-critical systems, including real-world driving. While stress is commonly detected using physiological signals such as perinasal perspiration and heart rate, facial activity provides complementary cues that can be captured unobtrusively from video. We propose a multimodal stress estimation framework that combines facial videos and physiological signals, remaining effective even when biosignal acquisition is challenging. Facial behavior is represented using a dense 3D Morphable Model, yielding a 56-dimensional descriptor that captures subtle expression and head-pose dynamics over time. To study how stress modulates facial motion, we perform extensive experiments alongside established physiological markers. Paired hypothesis tests between baseline and stressor phases show that 38 of 56 facial components exhibit consistent, phase-specific stress responses comparable to physiological markers. Building on these findings, we introduce a Transformer-based temporal modeling framework and evaluate unimodal, early-fusion, and cross-modal attention strategies. Cross-modal attention fusion of 3D-derived facial features with physiological signals substantially improves performance over physiological signals alone, increasing AUROC from 52.7% and accuracy from 51.0% to 92.0% and 86.7%, respectively. Although evaluated on driving data, the proposed framework and protocol may generalize to other stress estimation settings.