Machine vision detection to daily facial fatigue with a nonlocal 3D attention network

TL;DR

This paper introduces a novel audiovisual dataset and a 3D-ResNet based framework with non-local attention for detecting facial fatigue in real-world conditions, achieving high accuracy and capturing dynamic facial features.

Contribution

The paper provides a new daily-life fatigue dataset and a deep learning framework that effectively detects mild facial fatigue in unconstrained environments.

Findings

Achieved 90.8% accuracy on validation set

Proposed a combined loss function for fatigue prediction

Captured micro and dynamic facial features in real-world scenarios

Abstract

Fatigue detection is valued for people to keep mental health and prevent safety accidents. However, detecting facial fatigue, especially mild fatigue in the real world via machine vision is still a challenging issue due to lack of non-lab dataset and well-defined algorithms. In order to improve the detection capability on facial fatigue that can be used widely in daily life, this paper provided an audiovisual dataset named DLFD (daily-life fatigue dataset) which reflected people's facial fatigue state in the wild. A framework using 3D-ResNet along with non-local attention mechanism was training for extraction of local and long-range features in spatial and temporal dimensions. Then, a compacted loss function combining mean squared error and cross-entropy was designed to predict both continuous and categorical fatigue degrees. Our proposed framework has reached an average accuracy of 90.8% on validation set and 72.5% on test set for binary classification, standing a good position compared to other state-of-the-art methods. The analysis of feature map visualization revealed that our framework captured facial dynamics and attempted to build a connection with fatigue state. Our experimental results in multiple metrics proved that our framework captured some typical, micro and dynamic facial features along spatiotemporal dimensions, contributing to the mild fatigue detection in the wild.