Extending Machine Learning to Predict Unbalanced Physics Course Outcomes

TL;DR

This study improves machine learning classification of unbalanced physics course outcomes, especially for D and F grades, by optimizing the random forest algorithm and analyzing demographic impacts on prediction accuracy.

Contribution

It extends previous methods to better predict D and F outcomes in unbalanced datasets using random forests and demographic analysis.

Findings

Random forest with threshold adjustment increased D/F prediction accuracy to 46%.

Optimized models achieved 69% accuracy for C/D/F and 46% for D/F outcomes.

Prediction sensitivity varied across demographic groups, with higher accuracy for underrepresented minorities and first-generation students.

Abstract

Machine learning algorithms have recently been used to classify students as those likely to receive an A or B or students likely to receive a C, D, or F in a physics class. The performance metrics used in that study become unreliable when the outcome variable is substantially unbalanced. This study seeks to further explored the classification of students who will receive a C, D, and F and extend those methods to predicting whether a student will receive a D or F. The sample used for this work ($N=7184$) is substantially unbalanced with only 12\% of the students receiving a D or F. Applying the same methods as the previous study produced a classifier that was very inaccurate, classifying only 20\% of the D or F cases correctly. This study will focus on the random forest machine learning algorithm. By adjusting the random forest decision threshold, the correct classification rate of the D or F outcome rose to 46\%. This study also investigated the previous finding that demographic variables such as gender, underrepresented minority status, and first generation status had low variable importance for predicting class outcomes. Downsampling revealed that this was not the result of the underrepresentation of these students. An optimized classification model was constructed which predicted the D and F outcome with 46\% accuracy and C, D, and F outcome with 69\% accuracy; the accuracy of prediction of these outcomes is called "sensitivity" in the machine learning literature. Substantial variation was detected when this classification model was applied to predict the C, D, or F outcome for underrepresented demographic groups with 61\% sensitivity for women, 67\% for underrepresented minority students, and 78\% for first-generation students. Similar variation was observed for the D and F outcome.