Real-time estimation of overt attention from dynamic features of the face using deep-learning

TL;DR

This paper presents a deep learning approach to estimate student attention in real-time from facial movements in video, enabling online engagement monitoring without group reference data and addressing privacy concerns.

Contribution

The study introduces a lightweight deep learning model that predicts individual attention from facial features, functioning effectively without needing group reference data.

Findings

Model predicts attention with R^2=0.38 on unseen data.

Model estimates attention from individual facial movements without group reference.

The approach operates on client side, enhancing privacy.

Abstract

Students often drift in and out of focus during class. Effective teachers recognize this and re-engage them when necessary. With the shift to remote learning, teachers have lost the visual feedback needed to adapt to varying student engagement. We propose using readily available front-facing video to infer attention levels based on movements of the eyes, head, and face. We train a deep learning model to predict a measure of attention based on overt eye movements. Specifically, we measure Inter-Subject Correlation of eye movements in ten-second intervals while students watch the same educational videos. In 3 different experiments (N=83) we show that the trained model predicts this objective metric of attention on unseen data with $R^2$=0.38, and on unseen subjects with $R^2$=0.26-0.30. The deep network relies mostly on a student's eye movements, but to some extent also on movements of the brows, cheeks, and head. In contrast to Inter-Subject Correlation of the eyes, the model can estimate attentional engagement from individual students' movements without needing reference data from an attentive group. This enables a much broader set of online applications. The solution is lightweight and can operate on the client side, which mitigates some of the privacy concerns associated with online attention monitoring. GitHub implementation is available at https://github.com/asortubay/timeISC