Multimodal Machine Learning for Automated Assessment of Attention-Related Processes during Learning

TL;DR

This paper presents novel multimodal machine learning methods for automated, scalable, and in-the-wild assessment of attention-related processes like mind-wandering and engagement in educational settings, improving objectivity and generalizability.

Contribution

It introduces new computational approaches for detecting attention shifts and engagement using eye tracking, video, and physiological data, applicable across diverse learning environments.

Findings

Successful detection of mind-wandering and attention states.

Greater gaze synchronization among attentive online learners.

Robust detection of hand-raising as engagement indicator.

Abstract

Attention is a key factor for successful learning, with research indicating strong associations between (in)attention and learning outcomes. This dissertation advanced the field by focusing on the automated detection of attention-related processes using eye tracking, computer vision, and machine learning, offering a more objective, continuous, and scalable assessment than traditional methods such as self-reports or observations. It introduced novel computational approaches for assessing various dimensions of (in)attention in online and classroom learning settings and addressing the challenges of precise fine-granular assessment, generalizability, and in-the-wild data quality. First, this dissertation explored the automated detection of mind-wandering, a shift in attention away from the learning task. Aware and unaware mind wandering were distinguished employing a novel multimodal approach that integrated eye tracking, video, and physiological data. Further, the generalizability of scalable webcam-based detection across diverse tasks, settings, and target groups was examined. Second, this thesis investigated attention indicators during online learning. Eye-tracking analyses revealed significantly greater gaze synchronization among attentive learners. Third, it addressed attention-related processes in classroom learning by detecting hand-raising as an indicator of behavioral engagement using a novel view-invariant and occlusion-robust skeleton-based approach. This thesis advanced the automated assessment of attention-related processes within educational settings by developing and refining methods for detecting mind wandering, on-task behavior, and behavioral engagement. It bridges educational theory with advanced methods from computer science, enhancing our understanding of attention-related processes that significantly impact learning outcomes and educational practices.