Multimodal Engagement Analysis from Facial Videos in the Classroom

TL;DR

This study develops and evaluates computer vision methods to automatically assess student engagement from facial videos in classroom settings, demonstrating promising results and potential for educational research and teacher training.

Contribution

The paper introduces a multimodal approach combining facial expression and head pose analysis with deep learning classifiers for classroom engagement detection, extending prior work to real classroom environments.

Findings

Best classifiers achieved AUCs of .620 and .720 for Grades 8 and 12.

Score-level fusion improves engagement classification performance.

Personalized models with 60-second data improve accuracy by .084 AUC.

Abstract

Student engagement is a key construct for learning and teaching. While most of the literature explored the student engagement analysis on computer-based settings, this paper extends that focus to classroom instruction. To best examine student visual engagement in the classroom, we conducted a study utilizing the audiovisual recordings of classes at a secondary school over one and a half month's time, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement levels from faces in the classroom. We trained deep embeddings for attentional and emotional features, training Attention-Net for head pose estimation and Affect-Net for facial expression recognition. We additionally trained different engagement classifiers, consisting of Support Vector Machines, Random Forest, Multilayer Perceptron, and Long Short-Term Memory, for both features. The best performing engagement classifiers achieved AUCs of .620 and .720 in Grades 8 and 12, respectively. We further investigated fusion strategies and found score-level fusion either improves the engagement classifiers or is on par with the best performing modality. We also investigated the effect of personalization and found that using only 60-seconds of person-specific data selected by margin uncertainty of the base classifier yielded an average AUC improvement of .084. 4.Our main aim with this work is to provide the technical means to facilitate the manual data analysis of classroom videos in research on teaching quality and in the context of teacher training.