Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

TL;DR

This study evaluates the generalizability of SRL-aligned predictive models using digital learning traces across different computer science courses and universities, highlighting the importance of context in early at-risk student identification.

Contribution

It demonstrates that while early prediction of at-risk students is possible with digital traces, model robustness across different educational settings varies significantly.

Findings

Random Forest achieved highest in-sample accuracy.

Elastic Net generalized more robustly across different contexts.

Model calibration declined when applied across institutions with different at-risk rates.

Abstract

STEM dropout rates remain high at universities, particularly in computer science programs with theory-intensive courses. Digital learning environments now capture rich behavioral data that could help identify struggling students early, yet the generalizability of data-driven prediction models across courses and institutions remains uncertain. Guided by self-regulated learning (SRL) theory, this study analyzed multimodal digital-trace data from three undergraduate theoretical computer science courses (N1 = 137, N2 = 104, N3 = 148) at two universities. Weekly SRL-aligned digital-trace indicators were modeled using Elastic Net, Random Forest, and XGBoost to evaluate predictive performance over time and across settings, and model calibration both within and across courses. Early prediction of at-risk students was feasible, with SRL-related behaviors such as time management, effort regulation, and sustained engagement emerging as key predictors. While Random Forest achieved the highest in-sample accuracy, Elastic Net generalized more robustly across contexts. Out-of-sample accuracy and calibration declined between institutions with different base rates, underscoring the contextual nature of predictive analytics in higher education. These findings suggest that digital learning traces enable early identification of at-risk students within courses, but generalizing predictive models beyond their original context requires caution, particularly if the at-risk rates differ between contexts.