DeepFace-Attention: Multimodal Face Biometrics for Attention Estimation with Application to e-Learning

TL;DR

This paper presents a multimodal facial analysis method to estimate attention levels in e-learning, utilizing behavioral and physiological signals, and demonstrates superior accuracy on the mEBAL2 database.

Contribution

It introduces a novel ensemble approach combining facial features and physiological signals for attention estimation, optimized for e-learning applications.

Findings

Global features perform better with score-level fusion as temporal window increases.

Local features are more effective with neural network fusion methods.

The proposed method outperforms existing state-of-the-art on mEBAL2.

Abstract

This work introduces an innovative method for estimating attention levels (cognitive load) using an ensemble of facial analysis techniques applied to webcam videos. Our method is particularly useful, among others, in e-learning applications, so we trained, evaluated, and compared our approach on the mEBAL2 database, a public multi-modal database acquired in an e-learning environment. mEBAL2 comprises data from 60 users who performed 8 different tasks. These tasks varied in difficulty, leading to changes in their cognitive loads. Our approach adapts state-of-the-art facial analysis technologies to quantify the users' cognitive load in the form of high or low attention. Several behavioral signals and physiological processes related to the cognitive load are used, such as eyeblink, heart rate, facial action units, and head pose, among others. Furthermore, we conduct a study to understand which individual features obtain better results, the most efficient combinations, explore local and global features, and how temporary time intervals affect attention level estimation, among other aspects. We find that global facial features are more appropriate for multimodal systems using score-level fusion, particularly as the temporal window increases. On the other hand, local features are more suitable for fusion through neural network training with score-level fusion approaches. Our method outperforms existing state-of-the-art accuracies using the public mEBAL2 benchmark.