Inferring User Facial Affect in Work-like Settings

TL;DR

This study investigates inferring user facial affect in work-like settings using multimodal data and machine learning, highlighting the importance of context-specific datasets and segment-level spectral features for accurate prediction.

Contribution

It introduces a novel approach to predict dimensional affect in realistic work scenarios, emphasizing the role of context-aware data and spectral features.

Findings

Facial affect varies between non-working and working conditions.

Work-like context data improves prediction accuracy.

Segment-level spectral features enhance affect prediction.

Abstract

Unlike the six basic emotions of happiness, sadness, fear, anger, disgust and surprise, modelling and predicting dimensional affect in terms of valence (positivity - negativity) and arousal (intensity) has proven to be more flexible, applicable and useful for naturalistic and real-world settings. In this paper, we aim to infer user facial affect when the user is engaged in multiple work-like tasks under varying difficulty levels (baseline, easy, hard and stressful conditions), including (i) an office-like setting where they undertake a task that is less physically demanding but requires greater mental strain; (ii) an assembly-line-like setting that requires the usage of fine motor skills; and (iii) an office-like setting representing teleworking and teleconferencing. In line with this aim, we first design a study with different conditions and gather multimodal data from 12 subjects. We then perform several experiments with various machine learning models and find that: (i) the display and prediction of facial affect vary from non-working to working settings; (ii) prediction capability can be boosted by using datasets captured in a work-like context; and (iii) segment-level (spectral representation) information is crucial in improving the facial affect prediction.