From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

TL;DR

This paper develops and benchmarks models to predict student effort and progress in online learning, demonstrating improved accuracy over heuristics and providing insights for personalized tutoring.

Contribution

It introduces engagement forecasting as a supervised prediction task using ITS logs, establishing a reproducible benchmark and analyzing feature importance for explainability.

Findings

Feature-based models reduce MAE by 22-33% compared to heuristics.

Percentile heuristics tend to overpredict effort and progress.

Recent activity features mainly drive effort prediction, content difficulty influences progress.

Abstract

Sustained effort is essential for realizing the benefits of intelligent tutoring systems (ITS), yet many learners disengage or underuse available practice time. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week. Using interaction log data from 425 middle-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks. We show that these feature-based models reduce mean absolute error (MAE) by 22-33% relative to heuristic baselines, including fixed-percentile rules adapted from prior work in other behavioral domains. We find that percentile heuristics systematically overpredict, whereas feature-based models better track student practice trajectories across weeks. To support explainability, we analyze feature importance and ablations, revealing target-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner-state and content difficulty signals. Finally, in a semi-structured user interview case study with eight college tutors, we examine how tutors reasoned about system-generated predictive features when setting goals with students. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor-learner goal setting and timely instructional decisions.