Predicting Long-Term Student Outcomes from Short-Term EdTech Log Data

TL;DR

This paper demonstrates that machine learning models trained on the first few hours of student interaction data can effectively predict long-term academic outcomes, aiding early intervention efforts.

Contribution

It introduces a novel approach of using short-term EdTech log data to predict end-of-year student performance across diverse datasets.

Findings

Short-term log data (2-5 hours) predicts long-term outcomes effectively.

Predictive models can identify students at various performance levels.

Early usage data provides valuable signals for educational assessment.

Abstract

Educational stakeholders are often particularly interested in sparse, delayed student outcomes, like end-of-year statewide exams. The rare occurrence of such assessments makes it harder to identify students likely to fail such assessments, as well as making it slow for researchers and educators to be able to assess the effectiveness of particular educational tools. Prior work has primarily focused on using logs from students full usage (e.g. year-long) of an educational product to predict outcomes, or considered predictive accuracy using a few minutes to predict outcomes after a short (e.g. 1 hour) session. In contrast, we investigate machine learning predictors using students' logs during their first few hours of usage can provide useful predictive insight into those students' end-of-school year external assessment. We do this on three diverse datasets: from students in Uganda using a literacy game product, and from students in the US using two mathematics intelligent tutoring systems. We consider various measures of the accuracy of the resulting predictors, including its ability to identify students at different parts along the assessment performance distribution. Our findings suggest that short-term log usage data, from 2-5 hours, can be used to provide valuable signal about students' long-term external performance.