Predicting Performance During Tutoring with Models of Recent Performance

TL;DR

This paper introduces a new model that emphasizes recent student performance data to improve predictions of student success in tutoring systems, outperforming existing models in accuracy.

Contribution

The paper presents the Recent-Performance Factors Analysis model, which incorporates data recency and demonstrates superior predictive accuracy over prior models.

Findings

The new model outperforms existing logistic regression models.

Data recency significantly improves prediction accuracy.

Cross-validation with 0-1 loss is less effective than AIC and L1 loss.

Abstract

In educational technology and learning sciences, there are multiple uses for a predictive model of whether a student will perform a task correctly or not. For example, an intelligent tutoring system may use such a model to estimate whether or not a student has mastered a skill. We analyze the significance of data recency in making such predictions, i.e., asking whether relatively more recent observations of a student's performance matter more than relatively older observations. We develop a new Recent-Performance Factors Analysis model that takes data recency into account. The new model significantly improves predictive accuracy over both existing logistic-regression performance models and over novel baseline models in evaluations on real-world and synthetic datasets. As a secondary contribution, we demonstrate how the widely used cross-validation with 0-1 loss is inferior to AIC and to cross-validation with L1 prediction error loss as a measure of model performance.