Unobtrusive and Multimodal Approach for Behavioral Engagement Detection of Students

TL;DR

This paper introduces a multimodal, unobtrusive method for detecting students' behavioral engagement states in real classroom settings by combining appearance, context-performance, and mouse data.

Contribution

It presents a novel multimodal fusion approach for behavioral engagement detection using unobtrusive data sources in authentic educational environments.

Findings

High accuracy in engagement state classification

Effective fusion of multiple modalities improves detection performance

Validated on real classroom student data

Abstract

We propose a multimodal approach for detection of students' behavioral engagement states (i.e., On-Task vs. Off-Task), based on three unobtrusive modalities: Appearance, Context-Performance, and Mouse. Final behavioral engagement states are achieved by fusing modality-specific classifiers at the decision level. Various experiments were conducted on a student dataset collected in an authentic classroom.

Figures

Tables

Table 1. Table 1: F1-measures for uni-modal models (Appr: Appearance , CP: Context-Performance , Ms: Mouse ) and fusion (INSTR.: Instructional , ASSESS.: Assessment ).

Section Type	Class	Appr	CP	Ms	FUSION
INSTR.	On-Task	0.78	0.66	0.60	0.75
	Off-Task	0.71	0.50	0.51	0.64
	OVERALL	0.74	0.59	0.54	0.70
ASSESS.	On-Task	0.87	0.86	0.87	0.93
	Off-Task	0.66	0.22	0.64	0.73
	OVERALL	0.81	0.74	0.80	0.89

Full text

1 Introduction

Student engagement in learning is critical to achieving positive learning outcomes [4]. Fredricks et al. [7] framed student engagement in three dimensions: Behavioral, emotional, and cognitive. In this work, we focus on behavioral engagement, where we aim to detect whether a student is On-Task or Off-Task [10, 11] at any time of the learning task. Towards this end, we propose a multimodal approach for detection of students’ behavioral engagement states (i.e., On-Task vs. Off-Task), based on three unobtrusive modalities: Appearance, Context-Performance, and Mouse. Final outputs of behavioral engagement states are obtained by fusing modality-specific classifiers at the decision level.

2 Methodology

The proposed detection scheme incorporates data collected from three unobtrusive modalities: (1) Appearance: upper-body video captured using a camera; (2) Context-Performance: students’ interaction and performance data related to learning content; (3) Mouse: data related with mouse movements during the learning process. For a better evaluation of results, we analyzed the results separately for two learning tasks available: (1) Instructional, where students are watching videos; and (2) Assessment, where students are solving related questions.

Modality-specific data are fed into dedicated feature extractors [3, 9, 6], and the features are then classified with respective uni-modal classifiers (i.e., Random Forest Classifiers [5]). The decisions of separate classifiers are fused to output a final behavioral engagement state. For fusion, we propose to obtain a decision pool by incorporating all decision trees of modality-specific random forests and compute majority voting. This is equivalent to summing modality-specific confidence values and selecting the label with the highest confidence. Further details of the modalities, extracted features, and various fusion approaches we explored can be found in the full version of this paper [1].

3 Experimental Results

Through authentic classroom pilots, data were collected while the students were consuming digital learning materials for Math on laptops equipped with cameras. In total, 113 hours of data were collected from 17 9th graders for 13 sessions (40 minutes each), including the three unobtrusive modalities. For feature extraction, a sliding window of 8-seconds with 4-second overlap was utilized as in [8] for each modality. The collected data were labeled using HELP [2] by three educational experts. For final ground truth labels, the windowing was also applied over three label sets, which was followed by majority voting and validity filtering.

For the classification experiments, we divided each student’s data into 80% and 20% partitions, for training and testing, respectively. In order to reduce the effect of overfitting, we conducted leave-one-subject-out cross-validation and applied 10-fold random selection to balance training sets. The uni-modal and fusion results for different learning tasks (averaged over all runs and all student) are summarized in Table 1. As these results indicate, for Instructional sections, Appearance modality performs best (0.74) due to the lack of interactions necessary for the other modalities. For Assessment, fusing all modalities yields the best performance (0.89).

4 Conclusion

In summary, for behavioral engagement, we get relatively high results when only Appearance modality is used for Instructional sections whereas the fusion of all modalities yields better results in Assessment sections. The experiments also showed that it is beneficial to have context-dependent classification pipelines for different section types (i.e., Instructional and Assessment). In the light of these results, we can say that context plays an important role even when different tasks in the same vertical (i.e., learning) are considered.