Multimodality in Online Education: A Comparative Study

TL;DR

This paper advocates for a multimodal affect recognition system in online education, combining cues like posture, facial expressions, eye tracking, and verbal cues to better assess student understanding.

Contribution

It introduces a multimodal approach using weighted majority voting to integrate various cues for improved affect recognition in online classrooms.

Findings

Compared different machine learning models for each cue

Proposed a weighted voting system for model integration

Highlighted the importance of multimodal cues in online education

Abstract

The commencement of the decade brought along with it a grave pandemic and in response the movement of education forums predominantly into the online world. With a surge in the usage of online video conferencing platforms and tools to better gauge student understanding, there needs to be a mechanism to assess whether instructors can grasp the extent to which students understand the subject and their response to the educational stimuli. The current systems consider only a single cue with a lack of focus in the educational domain. Thus, there is a necessity for the measurement of an all-encompassing holistic overview of the students' reaction to the subject matter. This paper highlights the need for a multimodal approach to affect recognition and its deployment in the online classroom while considering four cues, posture and gesture, facial, eye tracking and verbal recognition. It compares the various machine learning models available for each cue and provides the most suitable approach given the available dataset and parameters of classroom footage. A multimodal approach derived from weighted majority voting is proposed by combining the most fitting models from this analysis of individual cues based on accuracy, ease of procuring data corpus, sensitivity and any major drawbacks.