Starting Seatwork Earlier as a Valid Measure of Student Engagement

TL;DR

This paper introduces and validates session-level measures of student self-regulation based on log data, demonstrating their reliability, predictive power for learning outcomes, and cross-system generalizability without additional instrumentation.

Contribution

It extends prior work by developing system-agnostic, session-level engagement measures from log data that predict learning outcomes and generalize across different educational platforms.

Findings

Session measures show high reliability (G > .75).

They improve prediction of final math scores beyond prior knowledge.

Measures generalize across different learning systems.

Abstract

Prior work has developed a range of automated measures ("detectors") of student self-regulation and engagement from student log data. These measures have been successfully used to make discoveries about student learning. Here, we extend this line of research to an underexplored aspect of self-regulation: students' decisions about when to start and stop working on learning software during classwork. In the first of two analyses, we build on prior work on session-level measures (e.g., delayed start, early stop) to evaluate their reliability and predictive validity. We compute these measures from year-long log data from Cognitive Tutor for students in grades 8-12 (N = 222). Our findings show that these measures exhibit moderate to high month-to-month reliability (G > .75), comparable to or exceeding gaming-the-system behavior. Additionally, they enhance the prediction of final math scores beyond prior knowledge and gaming-the-system behaviors. The improvement in learning outcome predictions beyond time-on-task suggests they capture a broader motivational state tied to overall learning. The second analysis demonstrates the cross-system generalizability of these measures in i-Ready, where they predict state test scores for grade 7 students (N = 818). By leveraging log data, we introduce system-general naturally embedded measures that complement motivational surveys without extra instrumentation or disruption of instruction time. Our findings demonstrate the potential of session-level logs to mine valid and generalizable measures with broad applications in the predictive modeling of learning outcomes and analysis of learner self-regulation.