Facial Movement Dynamics Reveal Workload During Complex Multitasking

TL;DR

This study demonstrates that facial movement dynamics captured by standard webcams can effectively monitor cognitive workload during multitasking, offering a low-cost, real-time alternative to traditional measures.

Contribution

It introduces a novel approach using facial keypoints and machine learning to assess workload, with models that adapt quickly and outperform existing performance metrics.

Findings

Facial movement features correlate with workload levels.

Participant-specific models achieve up to 73% accuracy.

Cross-participant generalization remains limited.

Abstract

Real-time cognitive workload monitoring is crucial in safety-critical environments, yet established measures are intrusive, expensive, or lack temporal resolution. We tested whether facial movement dynamics from a standard webcam could provide a low-cost alternative. Seventy-two participants completed a multitasking simulation (OpenMATB) under varied load while facial keypoints were tracked via OpenPose. Linear kinematics (velocity, acceleration, displacement) and recurrence quantification features were extracted. Increasing load altered dynamics across timescales: movement magnitudes rose, temporal organisation fragmented then reorganised into complex patterns, and eye-head coordination weakened. Random forest classifiers trained on pose kinematics outperformed task performance metrics (85% vs. 55% accuracy) but generalised poorly across participants (43% vs. 33% chance). Participant-specific models reached 50% accuracy with minimal calibration (2 minutes per condition), improving continuously to 73% without plateau. Facial movement dynamics sensitively track workload with brief calibration, enabling adaptive interfaces using commodity cameras, though individual differences limit cross-participant generalisation.