Improving state-of-the-art in Detecting Student Engagement with Resnet and TCN Hybrid Network

TL;DR

This paper introduces a novel hybrid neural network combining ResNet and TCN to improve the accuracy of detecting student engagement levels from videos, advancing the state-of-the-art in online learning analysis.

Contribution

The paper presents a new end-to-end ResNet+TCN architecture that effectively captures spatial and temporal features for engagement detection, outperforming existing methods.

Findings

Outperforms all compared methods on the DAiSEE dataset.

Sets a new baseline for student engagement detection accuracy.

Demonstrates the effectiveness of combining spatial and temporal neural networks.

Abstract

Automatic detection of students' engagement in online learning settings is a key element to improve the quality of learning and to deliver personalized learning materials to them. Varying levels of engagement exhibited by students in an online classroom is an affective behavior that takes place over space and time. Therefore, we formulate detecting levels of students' engagement from videos as a spatio-temporal classification problem. In this paper, we present a novel end-to-end Residual Network (ResNet) and Temporal Convolutional Network (TCN) hybrid neural network architecture for students' engagement level detection in videos. The 2D ResNet extracts spatial features from consecutive video frames, and the TCN analyzes the temporal changes in video frames to detect the level of engagement. The spatial and temporal arms of the hybrid network are jointly trained on raw video frames of a large publicly available students' engagement detection dataset, DAiSEE. We compared our method with several competing students' engagement detection methods on this dataset. The ResNet+TCN architecture outperforms all other studied methods, improves the state-of-the-art engagement level detection accuracy, and sets a new baseline for future research.