AI-Driven Early Warning Systems for Student Success: Discovering Static Feature Dominance in Temporal Prediction Models

TL;DR

This study analyzes temporal prediction models for student success, highlighting the dominance of static demographic features and demonstrating how model performance varies with intervention timing in online courses.

Contribution

It extends temporal analysis to Week 20, compares Decision Tree and LSTM models, and reveals static features' dominance in early prediction without needing ongoing data.

Findings

Static demographic features account for 68% of prediction importance.

LSTM achieves 97% recall at Week 2 for early intervention.

Decision Tree maintains 78% accuracy mid-course.

Abstract

Early identification of at-risk students is critical for effective intervention in online learning environments. This study extends temporal prediction analysis to Week 20 (50% of course duration), comparing Decision Tree and Long Short- Term Memory (LSTM) models across six temporal snapshots. Our analysis reveals that different performance metrics matter at different intervention stages: high recall is critical for early intervention (Weeks 2-4), while balanced precision-recall is important for mid-course resource allocation (Weeks 8-16), and high precision becomes paramount in later stages (Week 20). We demonstrate that static demographic features dominate predictions (68% importance), enabling assessment-free early prediction. The LSTM model achieves 97% recall at Week 2, making it ideal for early intervention, while Decision Tree provides stable balanced performance (78% accuracy) during mid-course. By Week 20, both models converge to similar recall (68%), but LSTM achieves higher precision (90% vs 86%). Our findings also suggest that model selection should depend on intervention timing, and that early signals (Weeks 2-4) are sufficient for reliable initial prediction using primarily demographic and pre-enrollment information.