Bag of States: A Non-sequential Approach to Video-based Engagement Measurement

TL;DR

This paper introduces a non-sequential, bag-of-words approach to measure student engagement from videos, challenging the necessity of modeling temporal order and achieving significant accuracy improvements over traditional sequential models.

Contribution

The paper proposes a novel non-sequential, bag-of-words-based model for engagement measurement, demonstrating its effectiveness over existing sequential and spatiotemporal models.

Findings

Improved engagement classification accuracy by 26% on IIITB dataset.

Achieved up to 66.58% accuracy on DAiSEE dataset.

Challenged the assumption that temporal order modeling is essential for engagement detection.

Abstract

Automatic measurement of student engagement provides helpful information for instructors to meet learning program objectives and individualize program delivery. Students' behavioral and emotional states need to be analyzed at fine-grained time scales in order to measure their level of engagement. Many existing approaches have developed sequential and spatiotemporal models, such as recurrent neural networks, temporal convolutional networks, and three-dimensional convolutional neural networks, for measuring student engagement from videos. These models are trained to incorporate the order of behavioral and emotional states of students into video analysis and output their level of engagement. In this paper, backed by educational psychology, we question the necessity of modeling the order of behavioral and emotional states of students in measuring their engagement. We develop bag-of-words-based models in which only the occurrence of behavioral and emotional states of students is modeled and analyzed and not the order in which they occur. Behavioral and affective features are extracted from videos and analyzed by the proposed models to determine the level of engagement in an ordinal-output classification setting. Compared to the existing sequential and spatiotemporal approaches for engagement measurement, the proposed non-sequential approach improves the state-of-the-art results. According to experimental results, our method significantly improved engagement level classification accuracy on the IIITB Online SE dataset by 26% compared to sequential models and achieved engagement level classification accuracy as high as 66.58% on the DAiSEE student engagement dataset.